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ABSTRACT 

The  purpose  of  systematic  evaluation  procedures  Is  to  help  policy 
analysts  confronted  with  difficult  decisions  to  fern  vise  choices.  Few 
useful  guidelines  have  been  offered  for  the  evaluation  of  action  alterna¬ 
tives,  especially  in  the  realm  of  foreign  policy  formation.  Schemes  based 
on  statements  of  goals  or  objectives  or  on  the  use  of  broadly  applicable 
criteria  for  choosing  among  alternatives  can  be  misleading,  and  they  are 
frequently  found  to  be  unworkable.  Rather  than  searching  for  particular 
rules  by  which  wise  choices  should  be  made.  It  is  more  useful  to  find 
ways  In  which  dll  'icult  decision  problems  can  be  decomposed  Into  smaller 
questions,  each  of  which  Is  easier  to  answer  than  the  larger  problem.  A 
tabular,  account-book  format,  described  as  the  revised  general  ledger . 
provides  a  sound  basis  for  systematically  comparing  and  evaluating  those 
features  which  significantly  differentiate  the  action  alternatives  under 
examination.  The  scheme  may  be  used  as  the  core  of  an  efficient  approach 
to  policy  analysis  described  as  the  palr-wlse  evaluation  strategy. 
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THE  EVALUATION  OF  POLICY  ALTERNATIVES1 

George  Kent 
University  of  Hawaii 


1.  INTRODUCTION 

i 

1.1  The  Problem 


The  purpose  of  this  essay  is  to  explore  the  character  of  evaluation 
as  it  occurs  in  the  work  of  policy  formation.  Evaluation  is  understood 
here  to  refer  to  the  task  of  determining  which  of  several  alternatives 
confronting  a  decision-maker  is  preferred  and  should  be  chosen.  The 
alternatives  are  assumed  to  be  concrete  action  possibilities ,  and  not 
grand,  abstract  sets  of  principles.  While  emphasis  is  placed  on  the 
problem  of  choosing  among  foreign  policy  alternatives,  many  of  the  observ¬ 
ations  that  are  made  will  apply  in  other  domains  as  well. 

The  term  evaluation  has  been  used  in  many  different  ways ,  as  will  be 
apparent  from  the  review  in  the  following  subsection.  Here,  the  problem  is 
deliberately  connected  with  decision-making:  given  that  a  choice  must  be 
made  between  two  or  more  possible  courses  of  action,  how  should  one  decide 
which  of  them  should  be  chosen?  In  this  view,  the  assessment  of  the  properties 
of  alternatives  is  instrumental  to  the  end  of  determining  preferences,  and 
is  part  of  the  larger  task  of  evaluation,  but  the  assessment  of  properties 
does  not  in  itself  constitute  the  whole  task  of  evaluation. 

The  problem  under  study  is  essentially  that  of  a  single  individual 
trying  to  decide  his  own  preferences.  Another,  different  problem  arises 


when  several  individuals,  in  a  common  committee  or  as  political  opponents, 
confront  each  other  with  Incompatible  preferences  and  try  to  reconcile  their 
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differences.  This  problem  of  group  decision-making,  vhere  the  individual 
members'  preferences  are  presumed  to  be  veil-specified  at  the  outset,  has  been 

studied  in  a  vide  variety  of  formats,  including  bargaining  and  gemlng,  and 

o 

also  in  the  studies  on  social  welfare  functions.  The  group  problem  is  set 
aside  here  on  the  grounds  that  the  formulation  of  preferences  by  the  individual 
is  naturally  prior  to  the  question  of  hov  several  different  decision-makers 
reconcile  their  preferences.  As  de  Rivera  put  it,  "in  the  last  analysis,  it 

•a 

is  the  conflict  within  each  person  that  will  decide  policy." 

Paul  Diesing  has  characterised  political  decisions  by  the  way  in  which 
they  are  made,  rather  than  in  terms  of  their  subject  matter: 


Non-political  decisions  are  reached  by  considering  a 
problem  in  its  own  terms,  and  by  evaluating  proposals 
according  to  bow  well  they  would  solve  the  problem. 

The  best  available  proposal  should  be  accepted  regardless 
of  who  makes  it  or  who  opposes  it,  and  a  faulty  proposal 
should  be  rejected  or  improved  no  matter  who  makes  it. 
Compromise  is  always  irrational',  the  rational  procedure 
is  to  determine  which  proposal  is  best,  and  to  accept  it. 

In  a  political  decision,  on  the  other  hand,  action  never 
is  based  on  the  merits  of  a  proposal  but  always  on  who 
makes  it  and  who  opposes  it.  Action  should  be  designed 
to  avoid  complete  identification  with  any  proposal  and  any 
point  of  view,  no  matter  hov  good  or  hov  popular  it  might 
be.  The  best  available  proposal  should  never  be  accepted 
Just  because  it  is  best;  it  should  be  deferred,  objected  to, 
discussed,  until  major  opposition  disappears.  Compromise 
is  always  a  rational  procedure,  even  when  the  compromise 
is  between  a  good  and  a  bad  proposal.1* 


This  study  is  not  about  the  politics  of  policy-making.  Political 
factors  do  greatly  influence  foreign  policy  decisions,  but  this  may  be 
precisely  because  there  is  no  well-developed  alternative  procedure.  A  major 
motivation  of  this  work  is  to  help  develop  practical  means  for  identifying 
wise  choices  which  are  better  than  the  essentially  arbitrary  means  of 
weighing  political  power  within  bureaucracies.  In  other  vords,  the  goal 


! 


3. 


Is  to  convert  political  decisions  into  non-political  ones.  Of  course, 
decision* on  matters  of  policy  must  take  full  account  of  likely  political 
consequences,  but  the  way  in  which  they  are  taken  into  account  should, 
in  the  sense  described  here,  be  apolitical. 

Unfortunately,  political  analysis  as  an  object  of  study  has  almost 
always  been  taken  to  refer  to  what  political  scientists  do,  and  not  to 

I 

what  political  actors  do.  As  a  result,  most  studies  about  political 
analysis  are  hopelessly  irrelevant  to  the  needs  of  actors. 

There  have  been  many  studies  of  how  people  do  arrive  at  their  decisions, 
both  in  political  and  in  non-political  contexts,  but  few  studies  have  been 
devoted  to  the  question  of  how  they  should  arrive  at  their  decisions.^  It  may 
be  that  people  do  tend  to  be  highly  responsive  to  peer  group  pressures,  or 
that  statesmen  are  not  much  concerned  with  moral  questions,  but  observations 
of  this  kind  tell  us  little  about  bow  they  should  respond.  This  study  is 
concerned  more  with  prescription  than  with  description  or  explanation.  How 
should  one  decide  what  action  to  take?  What  advice  can  be  offered  to  the 
policy  analyst  faced  with  a  difficult  choice?  What  analytical  procedure 
might  be  useful? 

Obviously,  the  answer  to  the  problem  of  evaluation  will  not  be 
provided  here.  There  is  no  right  answer,  but  some  answers  are  better 
than  others.  The  objective  of  this  study  is  to  raise  the  question,  to 
encourage  its  serious  study,  to  identify  some  of  the  common  difficulties, 
and  to  suggest  some  usable  procedures  for  the  work  of  evaluation.  More 
satisfactory  answers  will  ultimately  be  obtained  only  through  slow 
and  cumulative  efforts,  and  only  after  many  more  mistakes  have  been  made, 
both  in  analysis  and  in  action. 
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The  following  subsection  of  this  study  offers  a  sampling  of  the 
views  of  students  of  foreign  policy  on  the  problem  of  evaluation.  Rather 
than  responding  to  each  of  these  views  individually,  ray  critical  reactions 
are  implied  in  the  arguments  that  are  developed  later.  This  review  is 
followed  by  a brief  discussion  of  the  distinction  between  past-directed 
and  future-directed  evaluation. 

There  is  much  more  to  policy  analysis  than  Just  the  evaluation  of 
alternatives.  To  place  that  task  into  its  larger  context,  some  of  the 
other  phases  of  the  policy  formation  process  are  discussed  in  the  second  major 
section.  The  study  as  a  whole,  however,  focuses  on  the  problem  of 
choosing  among  alternatives,  and  not  on  general  policy  formation  or 
problem  solving  procedures. 

The  purpose  and  character  of  evaluation  procedures  are  discussed  in 
the  third  section.  This  section  reflects  personal  understanding  and  aspira¬ 
tion  more  than  empirical  description,  so  it  should  be  understood  as  being 
primarily  prescriptive  in  nature.  Naturally,  this  characterization  im¬ 
plicitly  anticipates  the  proposal  to  be  developed  in  the  conclusion. 

In  the  fourth  section,  I  review  a  variety  of  suggested  evaluation 
schemes.  I  conclude  with  a  description  of  my  favorite  system,  a  general 
ledger  accounting  fora  vhlch  seems  to  Incorporate  the  best  features  of  the 
other  procedures  that  have  been  suggested.  It  does  not  make  the  gross 
errors  of  other  formulations,  and  makes  up  in  practical  applicability  for 
what  it  lacks  in  apparent  sophistication.  Finally,  I  describe  a  policy 
analysis  strategy  suggested  by  the  preceding  arguments,  and  then  summarize 
the  major  observations  of  the  essay. 


Evaluation"  in  Foreign  Policy  Studies 

T 

Eugene  Meehan,  noting  thjt  there  are  certain  areas  of  behavior  "in 
which  the  consequences  of  choice  may  be  catastrophic  for  very  large  seg¬ 
ments  of  the  world's  population,"  asks  "How  can  such  choices  best  be 
made?  The  question  is  urgent : 


The  Justifications  offered  in  support  of  existing  values 
are  seldom  very  convincing.  And  there  is  little  evidence 
o  s  ow  that  the  quality  of  the  value  Judgments  men  make 
improves  with  the  seriousness  of  the  decisions  being 
made,  and  some  evidence  to  the  contrary-witness  the 
considerations  alluded  to  when  modern  governments  make 
major  policy  decisions.  According  to  the  news  media,  and 
to  the  participants  themselves,  stakes  of  unbelievable 
magnitude  are  wagered  out  of  irrational  fear,  foolish 
pride,  conceit,  ignorance,  malice,  and  even  stupidity. 

With  Which  the  “odists  of  the  age  attack 
°°re  than  Justified,  though  the  attack 
wouid  fare  better  if  the  reasoning  of  the  critics  were  ” 

better  than  th*  reasoning  of  those  they 
criticize— and  that  is  seldom  the  case.6 


The  need  for  research  on  bases  for  guiding  Judgment  in  national  and 
international  decision-making  was  described  by  Richard  Snyder  and  James 
Robinson  more  than  a  decade  ago: 


Value  Judgments  of  policy  decision  processes  are 
.natural  and  necessary,  but  rigorous  analysis  and  con¬ 
ceptual  clarification  of  criteria  have  been  notably 
lacking.  In  the  absence  of  explicit  and  agreed 
criteria,  the  evaluation  of  policy  consequences  gives 
ise  to  some  of  the  most  troublesome  problems  in  the 

£2*  "*??  0f  PUbllc  ,ffairs-  If.  *»  the  late  John 
.  *  ,  Du11m  ••••rttd,  ware  result  from  miscalculation, 
Js  an  error  in  policy-making  and  how  is  it  to  be 
ascertained  or  measured? ^ 


The  question  has  remained  unanswered,  and  worse,  ignored.  Prescrip- 
have  come  to  be  regarded  as  unscholar ly,  and  have  been  shunned 
y  udents  of  foreign  policy. °  One  result  has  been  that,  at  least  so  far 
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aa  the  development  of  potentially  useful  guidelines  is  concerned,  the  problem 
of  the  evaluatirn  of  foreign  policy  alternatives  has  received  very  little 
attention.  The  question  has  been  recognized  by  some,  of  course.  The  kinds 
of  answers  that  are  given  by  students  of  foreign  policy  can  be  illustrated 
with  a  few  examples. 

According  to  Glenn  Paige's  definition 


By  evaluation  is  meant  the  Judgment  (assignment  of 
values  to)  actual  or  potential  empirical  states  of 
affairs  in  terms  of  certain  criteria.* 


His  evaluative  method 


.  .  .  is  suggested  by  .  studies  of  "ordinary  language" 
tnat  have  been  done  by  ethical  theorists  in  philosophy. 
That  is,  rather  than  Impose  from  outside  a  set  of 
evaluative  criteria  that  has  been  created  out  of 
professional  polemics  in  political  science,  we  might 
begin  with  the  ordinary  language  of  moral  discourse 
that  is  revealed  by  the  Korean  case  materials 
themselves . 1 0 


Paige  analyzed  the  responses  of  critical  observers  of  the  United 
States'  decision  to  Intervene  in  Korea  in  1950  and  summarized  their  views 
in  eight  different  normative  propositions,  some  saying  the  decision  was 
wrong,  and  some  saying  it  was  right.  No  attempt  was  made  to  coalesce  the 
findings  into  a  single  encompassing  Judgment.  Certain  categories  and 
criteria  of  evaluation  were  suggested  by  the  analysis,  however.  These  were 


Antecedent  behavior :  Was  the  behavior  of  the  decision¬ 
makers  in  the  pre-crisis  period  such  as  to  minimize 
the  occurrence  of  the  crisis  precipitating  the  event? 
Decisional  process :  Was  the  response  to  the  crisis 
decided  in  such  a  way  as  to  gain  widespread  acceptance 
of  the  authority  of  the  decision  makers  through  the 
legitimacy  of  the  decisional  processes? 

Ends:  Were  the  ends  pursued  of  deep  and  enduring 
human  value? 

Means :  Were  the  means  employed* such  as  to  receive 
widespread  acceptance  as  being  ^appropriate  for  the 
ends  sought?11  \ 

'  \ 
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Paige  also  suggested  another  possible  approach: 


Another  avenue  for  exploration  in  evaluating 
the  Korean  decision  is  to  invoke  certain  criteria 
that  are  commonly  employed,  either  explicitly  or 
laplicitly  in  political  science  analyses.  This  approach 
differs  from  that  of  the  normative  inventory 
in  that  it  brings  to  bear  criteria  of  Jedgment  that 
originate  outside  the  Korean  case. 12 


These  criteria  were  illustrated  by  questions  about  such  things  as  the 
attainability  of  the  ends,  the  suitability  of  the  means,  the  timeliness  and 
flexibility  of  the  response,  and  the  accuracy  of  the  calculated  support. 

Speaking  of  the  critical  assessment  of  past  actions,  anothex'  student 
of  foreign  policy,  John  Lovell  states  that 


The  desirability  of  a  decision  or  action,  however,  is 
detensined  from  an  assessment  of  the  consequences  ... 
knovledgable  evaluation  is  invariably  dependent  upon 
adequate  explanation.  To  decide  whether  the  decision 
of  the  Truman  administration  to  send  troops  to  Korea 
was  good  or  bad,  it  is  essential  to  investigate  its 
consequences  ....  Actually,  assessment  of  a  decision 
or  act  requires  xiot  only  a  determination  of  the 
consequences,  but  also  a  comparison  of  the  actual 
consequences  with  the  probable  ones  if  an  alternative 
course  of  action  had  been  selected  ....  In  short,  what 
is  required  in  the  evaluation  of  past  decisions, 
programs ,  and  organisations  is  a  comparison  of  what 
was  with  what  might  have  been.13 


Paige  was  concerned  with  the  critical  assessment  of  past  decisions 
made  by  others,  a  task  which  may  be  differentiated  from  our  main  concern, 
the  future-oriented  assessment  of  contemplated  actions.  Lovell  makes  that 
distinction,  and  according  to  his  terminology *  one  evaluates  past  actions, 
but  prescribes  future  actions.  Similarly,  in  Lovell's  usage,  one  explains 
past  events,  but  predicts  future  events: 


The  relationship  between  prediction  and  prescription 
is  analogous  to  that  between  explanation  and  evaluation. 

. . .  Thus ,  sound  prescription  rests  upon  accurate  predic¬ 
tion,  just  as  knovledfable  evaluation  rests  upon  accurate 
explanation.  Moreover,  Just  as  assessment  of  a  past 
decision  requires  a  comparison  of  what  was  with  what 
sight  have  been,  assessaent  of  future  decisions  and 
actions  requires  one  to  compare  the  consequences  of  a 
prescribed  course  of  action  with  the  probable  consequences 
of  alternative  courses.14 


Later,  Lovell  asks 

Can  any  universally  acceptable  criteria  be  established 
that  could  serve  policy  makers  as  guldeposts  in  choosing 
among  various  intermediate  policy  objectives  and  smong 
policy  means  and  tactics,  or  that  could  serve  the  student 
of  foreign  policy  in  evaluating  the  choices  made  by  policy 
makersf  ....  our  quest  is  for  a  standard  of  evaluation  that 
can  be  applied  objectively  rather  than  subjectively.  That 
is,  we  are  searching  for  a  standard  by  which  each  of  us,  in 
apprdlsing  the  same  set  of  foreign-policy  goals  and  means, 
would  arrive  at  the  same  conclusions  as  to  which  goals  and 
means  were  good  and  which  were  bad.1* 


He  then  examines  three  candidate  criteria,  legal  norms,  moral 
principles,  and  "realism"  to  determine  whether  any  of  them  is  "both  universally 

h16 

acceptable  and  operationally  meaningful.  As  Lovell  understands  the  last 

of  these  criteria,  "realistic  goals  and  means  would  be  those  that,  in  the 

existing  pattern  of  demands  and  opportunities,  would  best  maintain  or  promote 

the  national  interest."  All  three  criteria  are  found  wanting  for  predictable 

reasons,  including  especially  their  ambiguity  and  their  questionable  applic- 

id 

ability  in  concrete  cases. 

Lovell  finds  that 


. . .  reasonable  men  can  and  do  disagree  about  what  is 
legal,  moral,  and  in  the  national  interest,  and  under 
what  conditions  various  norms  are  applicable.  Indeed, 
we  may  conclude  that  politics  is  a  struggle  not  only 
to  influence  particular  policy' outcomes  or  to  Influence 

/ 

/ 

/ 

j 
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the  allocation  of  costs  and  benefits  that  accrue  from 
the  policy  process;  politics  is  also  a  struggle  to 
influence  the  structuring  of  no ms  by  which  various 
policy  choices  are  Judged  to  be  legitimate  and  broadly 
acceptable . 1 9 

Many  observers  suggest  that  wisdom  should  be  socially  defined. 
According  to  Charles  Lindblom,  for  example,  "Agreement  on  policy  thus 
becomes  the  only  practicable  test  of  the  policy's  correctness.”  In 
a  context  which  qualifies  its  meaning  considerably,  Roger  Hilsman  has 
said  that  "The  test  of  policy  is  not  that  it  will  most  effectively 
accomplish  an  agreed-upon  value,  but  that  a  wider  number  of  people 
decide  to  indorse  it...."  Interpreting  this  as  a  prescription,  Joseph 
de  Rivera  disagreed,  and  responded  to  this  statement  by  saying  that 


While  we  might  call  the  above  the  "test  of  consensus," 
it  would  be  a  mistake  to  assume  that  this  is  the  only 
test  of  policy  and  that  policy  can  not  be  evaluated  from 
a  value-free  point  of  view.  For  the  reed  test  of  policy 
is  not  simply  that  a  number  of  people  think  it  will  help 
them  achieve  their  values.  Thus,  in  spite  of  the  fact 
that  policy  is  a-ratlonal,  we  may  ask  a  kind  of  reasonable¬ 
ness  of  it.  We  would  like  the  policy  to:  (1)  achieve  the 
goals  which  its  supporters  hoped  it  would  achieve,  and  (2) 
to  achieve  these  without  having  unanticipated  consequences 
that  damage  other  values  which  its  supporters  hold.  If  a 
policy  meets  these  criteria,  then  it  is  a  "good"  policy, 
whether  or  not  it  is  good  for  a  particular  individual  who 
may  not  have  supported  the  policy.  We  may  call  this  the 
"test  of  reasonableness."22 


Hone  of  these  suggested  tests  snea  Adequate.  While  many  students 


of  foreign  policy  gladly  say  which  particui 


;tr  policies  seem  to  them  to  b 


wise  or  unwise,  few  have  had  anything  to  saV  about  how  one  should  decide 
Those  who  have  recognised  the  question  have  not  agreed  in  their  answers. 
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They  hare  been  most  effectl-re  la  shoving  that  the  answer  to  the  question 
of  how  foreign  policy  alternatives  should  be  evaluated  is  not  at  all 
obvious. 

1.3  Past  versus  future  Evaluation 

Although  one  sight  not  like  his  labels t  Lovell  does  sake  a  useful 
distinction  between  assessments  made  for  past  decisions  and  assessments  made 
for  future  decisions.  There  may  be  important  differences  in  the  ways  in 
which  the  two  kinds  of  tasks  should  be  carried  out.  The  critical  analysis 
of  past  decisions  is  not  the  major  concern  of  this  study,  but  a  brief  compara¬ 
tive  look  at  it  may  help  to  sharpen  understanding  of  the  policy  analyst's 
evaluation  task. 

As  quoted  earlier,  Lovell  says  that  "To  decide  whether  the  decision 
of  the  Truman  administration  to  send  troops  to  Korea  was  good  or  bad,  it  is 
essential  to  investigate  its  consequences."  Similarly,  Paige  suggests  that 
it  may  be  valid  to  ask  whether  the  Korean  decision  really  reduced  the 
the  likelihood  of  future  wars,  as  Tyumen  had  hoped.  He  suggests  that  it 
may  be  valid  to  include  in  one's  evaluation  "an  assessment  of  the  objective 
consequences  of  behavior  whether  appreciated  by  the  decision  ackers  or 
not."23 

It  is  useful  to  distinguish  between  the  decision-maker  and  his 
decision,  on  the  one  hand,  and  the  resulting  action  and  consequences,  on 
the  other  hand.  Decisions  are  made  and  actions  are  taken  because  certain 
consequences  are  intended  and  expected,  but  because  of  the  inescapable 
imperfections  in  knowledge,  these  may  be  different  from  the  consequences 
that  are  actually  obtained.  Therefore,  looking  backward,  there  are  two 
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ve ry  different  kinds  of  Judgments  one  esc  stake,  one  on  the  merits  of  the 
decision,  and  one  on  the  merits  of  the  action. 

If  one's  purpose  is  to  assess  the  wisdom  of  a  decision-maker,  it  is 
unreasonable  to  base  that  assesmsent  on  the  actual  consequences  of  the  action. 

If  actual  outcomes  are  considered,  the  decision-maker  is  charged  or  credited 

I 

even  with  the  unforeseeable  consequences  of  his  action.  The  wisdom  of  a  past 

decision  should  be  assessed,  not  on  the  basis  of  the  actually  obtained 

consequences,  as  seen  retrospectively,  but  on  the  basis  of  the  best  prediction 

2k 

that  could  have  been  made  at  the  time  of  making  the  decision.  Contrary 
to  the  suggestion  of  Lovell,  Paige,  de  Rivera,  and  others,  evaluations  for 
past  and  future  decisions  should  not  be  different  in  this  respect.  One  may 
regret  decisions  that  have  turned  out  badly,  but  that  is  different  from 
Judging  them  to  have  been  made  unwisely. 

In  critically  analyzing  past  decisions  made  by  others,  one  must  ask 
by  whose  standards  the  action  is  to  be  assessed.  Should  it  be  judged  against 
the  decision-maker's  goals  or  the  critic's?  In  general,  the  critic  is 
obligated  to  assess  not  only  the  decision  but  also  the  quality  of  the  actor's 
values.  Efficient  murderers  are  not  praiseworthy.  The  policy  analyst, 
evaluating  the  alternatives  he  himself  confronts,  ultimately  has  only  his 
own  values  to  consider,  while  the  critic,  assessing  a  decision  made  by 
someone  else,  has  that  actor's  values  to  consider  as  well  as  his  own.  The 
policy  analyst's  task  is  therefore  somewhat  simpler  than  the  critic's.2^ 

If  one's  purpose  is  to  Judge  the  quality  of  an  action,  in  the  sense 
of  trying  to  decide  whether  one  should  regret  its  having  been  taken,  there  is 
no  reason  whatever  to  consider  the  analysis  made  by  the  decision-maker  at  the 
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Cine.  Ills  Intentions  and  expectations,  and  his  values  as  well,  are  all 
irrelevant  tr  this  judgment.  This  kind  of  retrospective  critical  analysis 
has  little  In  common  with  forward-looking  policy  analysis. 

Even  the  hindsight  required  for  retrospective  analysis  is  often 
blurred  because  the  Identification  of  the  consequences  of  past  actions 
can  be  extraordlnarilly  difficult.  Some  sort  of  explanation  (causal  analysis) 
is  needed  to  determine  which  events  should  be  charged  or  credited  to  a  given 
action.  History  does  not  reveal  its  alternatives.  It  will  often  be  unclear 
as  to  which  events  can  be  traced  to  the  particular  action,  and  which  would 
have  happened  anyway. 


2.  PRE-EVALUATION 

2 . 1  The  Beginnings  of  Policy  Analyses 

To  place  the  work  of  evaluation  into  context,  some  of  the  other 
phases  of  the  policy-formation  process  are  sketched  out  here.  Evaluation 
is  still  defined  ,  quite  narrowly,  as  the  task  of  determining  pre¬ 
ferences  among  action  alternatives. 

Some  decision  theorists  speak  of  the  evaluation  of  alternatives  as 
if  that  were  synonymous  with  the  work  of  policy-making.  They  begin  by 
specifying  alternatives  which,  by  their  definition,  are  taken  to  be  well- 
defined,  mutually  exclusive,  and  collectively  exhaustive.  Unfortunately, 
real  problems  do  not  begin  that  way.  They  do  not  begin  as  choices  between 
a  clear  this  and  a  clear  that,  but  as  questions  of  how  to  get  there,  or 
even  more  vaguely,  as  questions  of  what  should  be  done  about  some  thing. 

To  speak  as  if  well-defined  alternatives  were  always  Immediately  at  hand 
drastically  oversimplifies  the  work  of  policy-making.  There  is  always  a 
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great  deal  of  work  to  be  done  before  any  evaluation  is  needed,  including 
the  identification  of  things  that  could  be  done,  and  then  the  reformulation 
of  these  possibilities  into  alternatives. 

Although  the  analytical  tasks  faced  in  the  pre-evaluation  stages 
of  policy  formation  have  been  overlooked  in  the  decision-making  literature, 
they  are  nevertheless  very  real.  The  following  subsections  Isolate 
and  characterise  some  of  the  tasks  in  a  highly  stylized  way.  In  practice, 
of  course,  the  work  does  not  normally  proceed  according  to  any  such 
step-vise  program.  Where  difficulties  are  encountered,  however,  it 
may  be  useful  to  prescribe  procedures  like  those  outlined  to  the  policy 
analyst. 

2.2  Discovering  and  Inventing  Possibilities 

What  are  the  different  things  a  given  actor  could  do  in  a  given 
situation?  In  asking  what  could  be  done,  the  analyst  poses  the  question 
of  capabilities,  a  more  or  less  pure  question  of  fact.  The  evaluative 
question  is  temporarily  set  aside.  The  analyst  does  not  yet  ask  himself 
about  the  wisdom  of  the  possible  actions,  but  at  this  stage  seeks  only  to 
identify  them. 

The  action  possibilities  are  by  no  means  obvious  simply  upon  the 
naming  of  a  problem.  They  can  be  identified  only  through  intensive  study 
of  the  particular  situation,  the  surrounding  circumstances,  the  participants 
involved,  the  actor's  resources,  and  so  on.  While  some  generalisations  might 
be  made  about  action  possibilities  in  certain  classes  of  situations,  the 
policy  analyst  concerned  with  a  specific,  concrete  problem  must  engage  him¬ 
self  in  a  close  study  of  that  problem,  and  must  become  deeply  familiar  with 
it  and  its  context.  Only  then  will  he  be  able  to  suggest  reasonably  refined 
and  meaningful  action  possibilities. 


To  illuetrate  the  discussion*  suppose  that  the  policy  analysis  to  be 
performed  is  that  of  formulating  recommendations  for  United  States  action 
in  relation  to  the  problem  of  the  Palestinians.  After  a  preliminary 
study,  the  policy  analyst  might  draw  up  a  list  of  things  vhich  the  United 
States  could  do  and  which  would  be  related  to  that  problem.  The  list 
might  look  something  like  this: 


The  United  States  could: 

1.  encourage  programs  designed  to  advance  the  welfare 
of  the  refugees  in  the  camps; 

2.  encourage  enhanced  political  organisation  of  the 
Palestinian  groups; 

3.  decrease,  continue,  or  increase  the  funds  it  supplies 
to  the  refugees  through  the  United  Nations  Relief 
and  Works  Agency,  UNRWA; 

4.  attach  any  of  a  variety  of  political  conditions  to 
the  funds  it  supplies  through  UNRWA; 

5.  modify  its  policies  for  supplying  arms; 

6.  call  for  and  arrange  a  variety  of  different  kinds 
of  conferences  or  c  omissions  to  propose  actions 
dealing  with  the  refugee  problem; 

7.  liberalise  immigration  lavs  so  that  Palestinian 
refugees  could  take  up  residence  in  the  United  States 
more  easily; 

8.  have  Congress  pass  resolutions  publicly  advocating 
programs  favored  by  the  United  States; 

9.  curtail  training  of  military  officers  from  Arab  states 
in  the  United  States; 

10.  make  financial  support  of  the  American  University  of 
Beirut  contingent  on  its  systematic  support  of 
political  outcomes  favored  by  the  United  States; 

11.  offer  to  provide  American  troops  to  serve  as  a  buffer 
between  Israel  and  the  Palestinians,  or  to  police 
occupied  territories; 

12.  resume  efforts  at  working  in  collaboration  with  the 
Soviet  Union  to  determine  arrangements  vhich  would  be 
satisfactory  to  Israel,  the  Palestinians,  and  the 
Arab  states. 


Obviously,  many  other  possible  actions  could  be  added  to  this  list 
Even  as  it  stands,  the  length  of  the  list  could  be  multiplied  many  times 
by  permuting  it  through  a  variety  of  possible  conditional  statements  or 
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other  qualifications,  by  including  all  the  different  possible  mixtures 
of  these  actions,  and  by  counting  the  many  different  time  sequences  In 
which  they  might  be  mixed.  The  basic  elements  can  be  compounded  In  many 
different  ways. 

The  number  of  possibilities  which  might  be  considered  depends  on  the 

i 

level  of  specificity  or  abstraction  at  which  they  are  identified.  Those 
less  familiar  with  the  particular  problem  would  be  likely  to  use  grosser, 
more  abstract  categories,  suggesting,  say,  that  the  United  States  might  take: 
1.  political;  2.  economic;  or  3.  military  actions.  These  labels  can  be 
useful  If  they  are  understood  as  reminders  to  search  more  Intensively  within 
each  of  these  categories.  Even  possibilities  like  those  listed  above  are 
highly  abstract.  Each  Item  on  that  list  stands  for  or  "contains"  Innumerable 
possibilities. 

Possibilities  described  at  any  level  of  abstraction  may  be  Included 
In  the  list.  When  the  list  of  possibilities  Is  first  drawn  up.  It  should  be 
viewed  simply  as  an  Idea  net.  Ideas  for  action  can  be  listed  as  they  occur 
to  the  analyst,  without  concern  for  organisation  of  the  list  and  without 
concern  for  the  detailed  description  of  each  possibility.  At  the  beginning 
stages  of  analysis,  something  like  "action  through  the  United  Nations, "  might 
be  listed,  simply  as  a  reminder  to  look  Into  that  category  of  possibilities. 

2.3  Reformulating  Possibilities  Into  Alternatives 

It  would  be  premature  to  try  to  systematically  evaluate  the  enttles 
on  the  initial  list  of  possibilities  to  determine  which  of  them  Is  best  or 
to  determine  a  preference  ranking  for  them.  No  choice  problems  have  yet  been 
posed.  There  Is  no  merit  to  comparing  the  qualities  of,  say,  providing 
Increased  funds  to  UNRWA  with  the  qualities  of  liberalizing  immigration  laws 
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since  there  Is  no  intrinsic  need  to  choose  between  these  possibilities.  They 
are  not  alternatives.  In  the  sense  that  they  are  not  mutually  exclusive; 

doing  one  of  these  things  does  not  foreclose  the  possibility  of  doing  the 

* 

other.  Actions  which  may  at  first  seem  to  be  mutually  exclusive  often  are 
not,  and  can  be  mixed  in  different  degrees,  if  not  at  the  same  time,  then  at 
least  through  time.  One  can  have  his  cake  and  pat  it,  at  different  times, 
or  at  the  seme  time  if  one  can  be  satisfied  with  eating  less  than  the  whole 
cake.  It  is  a  serious  mistake  to  insist  that  one  must  choose  between  one 
possible  action  or  another  when  in  fact  independent  choices  can  be  made  for 
each  of  them.  The  policy  analyst  should  not  suggest  a  choice  is  necensary 
where  in  fact  none  is  required  because  doing  so  would  waste  the  analyst's 
limited  resources,  and  more  importantly,  because  it  would  lead  to  the 
rejection  of  possible  action  mixtures  without  their  being  recognized  and 
examined . 

Thus,  no  matter  how  exhaustive  they  may  be,  lists  of  action  possib¬ 
ilities  need  reformulation  to  show  where  choices  must  be  made. 

It  Is  possible  to  formulate  alternatives  which  are  collectively  ex¬ 
haustive  and  mutually  exclusive  by  making  purely  logical  disjunctions,  that 
is,  by  asserting  that  a  particular  effort  must  be  increased  or  decreased  or 
allowed  to  remain  at  the  same  level,  or  by  counterpolnting  eacu  proposal 
with  Its  negation.  Sets  of  alternatives  like  the  following  might  be  des¬ 
cribed: 


The  United  States  could : 

1A.  encourage  programs  to  advance  the  welfare  of  the 
refugees  In  the  camps; 

OR  IB.  not  encourage  programs  to  advance  the  welfare  of 
the  refugees  In  the  camps. 
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The  United  States  could: 

3A.  decrease  the  funds  It  supplies  to  the  refugees 
through  the  UNRWA; 

OR  3B.  continue  the  funds  It  supplies  to  the  refugees 
through  the  UNRWA; 

OR  3C.  Increase  the  funds  it  supplies  to  the  refugees 
through  the  UNRWA. 

In  a  formal  sense,  choices  do  have  to  be  made  from  each  of  these 
sets  of  alternatives.  These  sets  do  not  bring  out  the  real  problems  faced  by 
the  policy  analyst,  however,  because  the  proposals,  as  stated,  are  much 
too  vague.  How  should  the  United  States  encourage  what  programs  to 
advance  the  welfare  of  the  refugees?  In  what  degree  should  the  United 
States  decrease  or  increase  appropriations  to  UNRWA?  Should  the  funds  be 
given  unconditionally?  Each  of  the  abstract  proposals  listed  earlier  can 
be  manifested  concretely  in  hundreds  of  different  ways,  and  choices  must 
be  mads  among  these  possible  variations.  If  an  affirmative  answer  is  given 
to  the  question  of  whether  to  encourage  welfare  programs,  one  must  still 
ask  what  kind  of  program  it  is  to  be,  and  this  in  turn  will  lead  to  many 
other  questions. 

This  suggests  a  hierarchical  decision-making  system  in  which  commit** 
ments  are  made  at  higher  levels  of  generality,  at  the  "strategic"  level, 
and  lower  level  "tactical"  questions  are  then  dealt  with  after  those  higher 
level  questions  have  been  settled.  That  procedure  is  simple  and  elegant, 
but  unrealistic.  It  creates  a  misleading  illusion  of  simplicity.  The 
higher  level  question  cannot  be  answered  In  its  abstract  form  without 
reference  to  the  way  In  which  the  concrete  action  that  is  suggested  will  be 
carried  out.  A  firm  commitment  cannot  be  made  at  the  higher  level  until 
some  of  the  lower  level  options  are  recognized  and  decided.  The  analyst 
can  pursue  his  work  by  asking,  the  United  States  were  to  encourage 
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welfare  programs  for  the  refugees,  what  form  should  these  programs  take? 

He  could  then  go  on  to  suggest  concrete  programs. 

It  should  be  appreciated  that  each  of  these  propoeed  programs  could 

then  be  evaluated  on  Its  own  merits.  Thera  would  be  no  need  to  go  back  up 

the  abstraction  ladder  to  answer  the  highly  generalized  question  about 

whether  that  kind  of  activity  should  be  undertaker.  Decision  problems 

posed  at  high  levels  of  generality  are  not  only  difficult  to  answer,  but 

are  often  not  worth  answering.  This  Is  Illustrated  by  a  series  of  studies 

that  v;ri  supposed  to  systsmatlcally  sz amine  the  problem  of  choosing  msong 

26 

five  different  basic  defense  postures.  The  authors  failed  to  daal  with 
the  substance  of  the  problem,  partly,  It  seems,  because  the  five  options  that 
were  offerad  were  never  made  concrata.  The  first  phase  of  their  analysis 
should  have  been  devoted  to  specifying  In  detail  how  each  of  the  five  con¬ 
figurations  would  have  been  Implemented  If  adopted.  Doing  this  would 
have  also  helped  to  show  whether  the  original  question  was  well  formed,  since 
that  Initial  probe  might  have  revealed  that  there  was  actually  a  smaller 
or  larger  variety  of  distinctly  different  dafsnsa  postures  which  needed 
examination. 

The  reformulation  of  a  list  of  possibilities  yields  a  large  number  of 
sets  of  alternatives.  Typically,  each  set  will  show  a  variety  of  ways  In 
which  a  given  kind  of  action  might  be  taken,  and  Include  a  null  altarnatlva 
saying  that  that  kind  of  action  should  ba  rejected  altogether.  One  choice 
must  be  made  from  each  set.  The  choices  from  the  different  eats  could  ba 
made  Independently  (by  definition),  but  the  analyst  Is  free  to  couple  some 
of  the  choices  together  or  to  condition  some  choices  on  others  In  any  way. 

For  many  of  thesa  sets  of  alternatives  tha  choices  will  not  be 
problematical,  and  It  will  ba  clear  which  of  the  alternatives  Is  tha  bast 
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of  a  given  set.  In  some  of  these  cases,  that  clearly  favored  option  will 
be  the  null  alternative,  the  one  that  says  one  should  forget  about  taking 
any  of  the  possible  variations  of  a  particular  kind  of  action.  Thus,  these 
dusters  of  action  possibilities  are  rejected.  In  other  obvious  cases, 

certain  positive  actions  will  be  seen  as  clearly  worth  pursuing. 

' 

The  policy  analyst  can  than  work  at  evaluating  the  alternatives  In 
those  sets  In  which  the  decision  Is  problematic,  or  he  can  develop  and 
pursue  those  positive  actions  which  appear  to  be  clearly  worth  pursuing. 
This  last  option  successfully,  and  parhapa  wisely,  evades  or  defers  the 
necessity  for  engaging  in  tha  work  of  systematic  evaluation.  It  may  be 
wiser — that  Is,  more  worthwhile,  for  the  policy  analyst  to  follow  up  on 
actions  clearly  worth  taking  than  to  expend  his  enargy  on  making  difficult 
choices . 

Where  he  does  find  It  difficult  to  choose  among  alternatives,  that 

i 

may  be  because  there  la  not  much  differences  among  the  altarnatlvas,  and 
the  analyst  knows  It,  or  becauss  hr  doaa  not  know  very  much  about  the 
alternatives.  In  the  first  kind  of  case,  ha  might  be  bettar  off  just  flip¬ 
ping  a  coin,  arbitrarily  choosing  one  or  another  of  the  nearly  equivalent 
alternatives.  In  the  second  kind  of  casa,  If  the  stakes  are  high  anough 
and  If  he  has  the  time  and  other  resources  to  do  so,  he  may  wish  to  Invest 
energy  Into  obtaining  more  Information  about  the  character  of  tha  action 
alternatives  and  about  their  likely  consaquences.  This  new  Information 
may  then  simplify  what  had  previously  been  a  difficult  choice. 

In  any  case,  the  policy  analyst  should  not  spread  his  attention 
uniformly,  and  thinly,  across  all  of  the  clusters  of  action  possibilities 
of  which  he  Is  able  to  conceive.  At  some  stage  he  will  have  to  focus  his 
attention  on  certain  ones,  and  set  aside  others.  Since  it  is  usually  not 
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worthwhile  to  conduct  a  thorough  "mats-"  policy  analyaia  to  decide  which 
of  them  is  most  worth  pursuing,  the  choice  will  have  to  ba  somewhat  arbit¬ 
rary.  The  analyst  should,  of  course,  use  his  best  Judgment  to  pick  out 
the  most  promising-appearing  possibilities. 

Thus,  even  though  there  may  not  be  any  external  constraints  forcing 
a  choice  among  certain  kinds  of  action  possibilities,  the  analyst's  own 
resource  limitations  may  force  him  to  choosa  among  them,  lie  should  be  care¬ 
ful  to  be  clear  about  which  possibilitias  are  rejected  because  his  analysis 
leads  to  the  conclusion  that  they  are  deficient,  and  which  are  simply  set 
aside  without  examination.  If  he  redefines  his  mandate  and  focuses  the 
body  of  his  analysis  on  just  one  cluster  of  possibilities,  on  funding 
policy  for  UNRWA,  for  example,  he  is  no  longer  asking  what  is  the  bast 
thing  the  United  States  could  do  about  the  problem  generally,  lie  should 
be  dear  that  his  concluding  recommendations  on  conditions  to  be  attached 
to  UNRWA  appropriations  are  not  to  be  understood  as  implying  a  rejection 
of  possible  actions  in  other  domains.  His  conclusions  should  acknowledge 
whatever  limitations  he  has  imposed  on  the  analysis,  and  should  Include 
a  clear  description  of  the  way  in  which  he  may  have  redefined  his  original 
problem. 

The  policy  analyst  has  now  arrived  at  a  stage  at  which  ha  can  lnvast 
his  effort  into  developing  and  implementing  the  alternatives  which  he 
has  already  chosen,  or  into  making  choices  from  those  sets  of  alternatives 
in  which  it  is  not  obvious  which  alternative  is  best.  Although  it  may  ba 
wisest  to  emphasize  the  first  of  these  courses,  the  discussion  hereafter 
will  focus  on  the  second.  A  varlaty  of  rough  and  intuitive  evaluations 
had  presumably  bean  mada  before  this  point,  but  it  is  only  at  this  staga 
that  the  policy  analyst  takes  up  and  squarely  confronts  the  problem  of 
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systematically  evaluating  alternatives.  The  discussion  of  the  task  of 

evaluation.  Itself,  begins  Immediately  after  the  following  subsection. 

\ 

2.4  Risk  Balancing 

Some  might  argue  that  the  list  of  possible  actions  to  be  considered 
should  be  exhaustive,  that  It  should  Include  all  action  possibilities. 

Taken  literally,  that  prescription  Is  absurd.  It  cannot  be  fulfilled  simply 
because  the  number  of  possible  actions  Is  limitless,  and  because  the  analyst’s 
resources  are  always  limited.  It  Is  reasonable,  however,  to  ask  that  there 
be  no  arbitrary  or  unjustified  conscious  exclusions. 

An  overly  long  list  would  be  unmanageable,  and  not  very  useful  snyway. 
The  policy  analyst's  art  lies,  In  part.  In  his  finding  good,  convenient,  and 
reasonable  ways  to  delimit  the  list  of  possibilities  to  be  examined. 

The  extent  to  which  It  Is  worthwhile  for  the  analyst  to  continue 
listing  possible  actions  depends  on  how  fruitful  that  effort  Is  likely  to  be 
when  compared  with  the  opportunities  for  Investing  that  same  amount  of  effort 
elsewhere  In  the  policy  snalysls.  Although  It  was  suggested  earlier  that 
the  analyst  should  be  unconcerned  with  the  quality  or  wisdom  of  the  sctlons 
when  he  draws  up  the  list  of  possibilities,  the  llmltstlons  on  the  analyst's 
resources  require  that  this  statement  be  softened.  A  rough.  Intuitive  eval¬ 
uation  must  be  made  just  to  determine  whether  It  Is  worthwhile  to  include 
any  given  action  possibility  on  the  list.  Because  of  the  limitations  on 
the  analyst’s  resources,  a  possibility  must  merit  further  examination.  It 
does  so  only  by  having  some  significant  chance  of  being  adopted.  Action 
possibilities  whose  detailed  examination  would  obviously  waste  the  analyst's 
time  should  be  excluded  as  early  as  possible.  Of  course,  there  is  a  risk 
of  premature  rejection  In  this  procedure.  Given  the  Inevitable  limitation 
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on  Che  resources  available  Co  be  devoced  Co  any  analysis,  that  risk  is 
inescapable . 

The  list  of  possibilities  can  be  extended  by  developing  ideas  for 
new  icems  to  be  added  to  Che  list,  or  by  intensively  examining  entries  in 
che  current  list  and  showing  Che  options  contained  within  them.  Efficient 
policy  analyses  commonly  begin  with  relatively  abstract  characterizations 
of  the  action  possibilities,  and  then,  as  some  ere  discarded,  energy  is 
Invested  into  refining  the  remaining,  promising  possibilities. 

Some  of  the  refinement  work  may  take  place  as  the  possibilities  ere 
identified,  and  some  nay  be  performed  after  alternatives  are  formulated 
and  it  becomes  clearer  which  possibilities  ere  worth  refining.  The  same 
sort  of  principles  apply  in  moving  up  and  down  the  abstraction  ladder  when 
dealing  with  sets  of  alternatives.  While  it  is  not  true  that  a  firm  commit¬ 
ment  must  be  maue  to  selecting  from  a  given  category  of  actions  before  the 
members  of  that  category  can  be  subjected  to  closer  scrutiny,  a  tentative 
commitment  does  have  to  be  made  to  determine  whether  it  is  worthwhile  to 
invest  energy  into  working  out  the  lower  level  options.  Again,  a  given 
cluster  of  possible  actions  must  merit  further  examination  by  holding  out  a 
substantial  promise  of  ultimately  being  chosen.  It  is  usually  not  worth¬ 
while  to  work  out  tactics  for  strategies  that  have  very  little  chance  of 
27 

being  implemented. 

It  is  possible  to  be  more  precise  about  how  finely  the  alternatives 
must  be  described.  Clearly,  they  must  be  developed  to  some  extent,  but  it 
is  also  clear  that  they  do  not  need  to  be  fully  developed,  down  to  fine 
details  of  Implementation.  The  analyst's  resources  must  be  Invested  carefully. 
For  the  purposes  of  policy  analyses,  alternatives  need  to  be  specified  only 
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to  the  degree  that  la  needed  to  make  a  clear  choice.  Detail  beyond  that 
level  can  be  decided  upon  after  particular  alternatives  are  chosen. 

The  policy  analysis  process  Involves  much  more  than  just  evaluation, 
and  those  other  steps  demand  the  analyst's  attention  as  veil.  The  purpose 
of  evaluation  la  to  help  form  choices.  There  Is  therefore  no  need  to  expend 
great  amounts  of  energy  on  evaluation  until  and  unless  It  Is  clear  what 
choices  need  to  be  made. 

3.  THE  PURPOSE  AND  CHARACTER  OP  EVALUATION  PROCEDURES 

3.1  The  Function  of  Evaluation  Procedures 

Since  the  function  of  evaluation,  as  defined  here.  Is  to  determine 
preferences  among  alternatives,  evaluation  must  be  understood  as 
fundamentally  comparative,  and  value  must  be  understood  as  fundamentally 
relativistic.  As  Eugene  Meehan  put  It 

...  If  a  value  judgment  Is  an  expression  of 
preference,  a  proposed  value  judgment  that  specifies 
only  one  outcome  Is  Improper  and  Incomplete,  whatever 
the  situation  to  which  It  epplles,  because  the  term 
"preference"  Implies  comparison  and  comparison 
requires  two  or  more  dess  members.28 

The  decision-maker  needs  to  know,  not  whether  a  given  action  would  be  good 
or  wise  In  an  absolute  sense,  but  whether  It  would  be  better  or  wiser  than 
Its  competitors. 

Since  his  purpose  is  simply  to  make  a  choice,  the  analyst  may  not 
be  much  concerned  with  how  much  better  one  alternative  is  than  another. 

If  he  does  not  need  that  Information,  he  may  not  find  It  worthwhile  to  pay 
the  extra  effort  required  to  determine  that  subtle  measure  of  degree.  It 
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may  be  sensible  for  him  to  rest  with  simple  determinations  of  superiority 
and  inferiority. 

The  purpose  of  evaluation  la  to  determine  preferences.  The  purpose 
of  systematic  eveluation  procedures  is  to  guide  the  determination  of  prefer¬ 
ences  in  problematic  situations,  that  is,  where  choices  are  difficult  to 
make.  Techniques  of  this  kind  are  of  no  uss  whore  the  decision-maker  is 
already  confident  in  his  selection  of  one  or  another  of  the  alternatives. 

He  might  use  the  technique  to  sanctify  decisions  ho  has  already  made,  and 
to  persuade  others  of  thslr  virtue,  but  that  use  would  be  an  abuse. 

If  it  were  possible  to  find  a  procedure  by  which  it  was  possible  to 
evaluate  alternetlves  without  the  analyst's  active  intervention,  that  pro¬ 
cedure  would  serve  as  a  substitute  for  Judgment,  and  would  also  allow  the 
analyst  to  remain  quite  Ignorant  of  the  nature  of  the  political  problem  at 
issue.  The  simplest  such  procedure,  flipping  a  coin,  requires  only  a  purely 
mechanical  activity,  end  does  not  demand  any  knowledge  or  Judgment  on  the 
part  of  the  "analyst."  Deferring  decisions  to  a  group  consensus  or  to  a 
more  or  less  arbitrary  voting  procedure  is,  slmllerlly,  a  way  of  not  deciding 
herd  Issues  on  their  merits.  It  is  possible  to  find  ways  to  avoid  facing 
up  to  a  decision,  but  no  really  good  way  has  yet  been  devised. 

Evaluation  procedures  of  the  kind  studied  here  are  Intended  to  help 
the  enalyst  to  deal  with,  rather  than  evade,  decisions.  They  should  be 
viewed  es  elds  to  judgment,  not  as  substitutes  for  Judgment.  They  demand 
deep  knowledge  of  the  particular  political  problem  and  of  the  alternatives 
at  hend. 

Three  major  kinds  of  systematic  evaluation  procedures  will  be  dis¬ 
cussed  in  Section  4,  the  use  of  goals  and  objectives,  the  use  of  criteria, 
and  the  decomposition  of  problematic  decisions.  None  of  these  procedures 
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eliminates  the  work  of  evaluation  and  the  need  for  judgment  and  decisions. 

They  only  change  the  scope  or  magnitude  or  locus  of  that  work  in  some  sense. 
Where,  for  example,  goals  or  objectives  are  offered  as  the  basis  for  choice, 

i 

those  goals  or  objectives  always  must,  in  turn,  be  selected  from  a  large 

i 

variety  of  alternative  goals.  If  a  certain  criterion  is  proposed,  the 

analyst  must  still  decide  that  being  higher  on  a  particular  measure  is  to 

be  preferred.  (Of  course,  seeking  criteria  by  which  to  select  criteria 

can  lead  to  the  same  endless  regression  encountered  In  selecting  goals.) 

Similarly,  decomposing  complex  alternatives  into  simpler  components  still 

requires  evaluative  judgments  on  each  of  those  components.  In  the  expected 

value  model,  for  instance,  where  actions  are  evaluated  according  to  the 

qualities  of  the  outcomes  to  which  they  might  possibly  lead,  those  possible 

outcomes  still  require  evaluation.  For  the  Incrementalists,  too,  "Choice 

among  policies  Is  made  by  ranking  In  order  of  preference  the  Increments  by 
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which  social  states  differ."  That  Is,  the  analyst  is  asked  slnply  to 
exercise  his  judgment  to  decide  which  marginal  alternative  would  be  best. 

The  evaluation  task  cannot  be  wholly  evaded,  but  It  can  be  rearranged  in 
such  a  way  that  It  becomes  more  manageable.  That  Is  the  function  of  a 
systematic  evaluation  procedure. 

3*2  Separating  the  Fact  and  Value  Questions 

Decisions  are  based  on  the  factual  and  the  value  aspects  of  the 
analyst's  understanding  of  the  alternatives  under  examination.  It  is  not 
only  the  value  component  which  is  "psychological"  and  Internal  to  the 
decision-maker,  while  the  relevant  facts  are  somehow  "objective"  and  external. 
The  factual  component  (past,  present,  and  projected)  which  enters  Into  the 
decision  Is  a  matter  of  judgment,  just  the  value  component.  The 
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decision  Is  necessarily  based  on  Che  way  Che  alcernaclves  appear  Co  Che 
decision-maker,  according  eo  his  best  Judgment  of  Che  evidence.  Ha  doss 
noC  compare  accual  consequences  or  acCual  coses  or  accual  features  of  any 
kind;  he  compares  his  informed  judgments  on  these  dimensions.  The  decision¬ 
maker  or  analyse  may,  of  course,  adopt  his  information  from  soma  authorita- 

j 

Clve  source,  but  it  Is  still  his  judgment,  rather  Chan  Che  external  fset, 

chat  enters  Into  the  analysis.  He  makes  his  choices  on  the  basis  of  his 
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beliefs  about  the  nature  of  the  alternatives  under  examination. 

Evaluative  Judgments  are  about  beliefs  of  feet.  It  follows  that 
difficulties  in  forming  evaluations  may  be  partielly  due  to  uncertainties 
with  respect  to  those  facts.  Difficulties  may  sometimes  be  encountered  even 
when  the  relevant  facts  seem  clear  to  the  decision-maker.  The  analytical 
process  involved  In  the  'pure"  evaluation  problem  can  be  raised  in  clear 
relief  by  separating  it  from  the  assocleted  questions  of  fact.  The  empirical 
and  evaluative  tasks  can  be  separated  by  postulating  a  variety  of  hypothetical 
answers  to  the  empirical  questions  that  arise,  and  then  performing  evalua¬ 
tions  and  determining  the  appropriate  choice  for  each  hypothetical  set  of 
facts.  The  usefulness  of  this  procedure  was  appreciated  by  Harold  Lasswell, 
especially  In  relation  to  projected  facts: 

It  Is  generally  held  that  the  evaluation  of  policy 
alternatives  is  more  a  field  of  professional  competence 
chan  that  of  the  making  of  comprehensive  estimates  of 
the  future.  A  specialist  can  make  himself  comfortable 
among  alternatives  of  policy  by  the  expedient  device 
of  adopting  successive  sets  of  working  assumptions 
about  future  contingencies.  The  expert  can  in  this 
wsy  evade  the  responsibility  for  committing  himself 
to  a  definite  estimate  of  the  likelihood  that  a  given 
contingency  will  in  fact  appear.31 
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Strictly  speaking,  purely  evaluative  analyses  should  always  report 
their  results  on  a  conditional  basis.  Rather  than  simply  saying  that  a 
certain  alternative  should  be  chosen,  the  findings  should  say  something 
to  the  effect  that  JLf  the  postulated  facta  ara  the  case  and  if  one  has  the 

i 

values  described,  then  one's  best  choice  would  be  to  take  such  and  such 
action.  If  the  underlying  argument  is  sound,  denial  of  these  stated 
conditions  can  then  provide  a  basis  for  denying  the  consequent  Imperative. 

Of  course,  in  practical  policy  analyses  It  may  be  awkward  or  impossible  to 
frame  concluding  recommendations  In  this  logical  form,  but  the  principle 
should  be  fully  appreciated,  and  the  body  of  the  argument  should  make 
clear  the  conditions  and  assumptions  on  which  the  final  recommendations 
are  based. 

One  Important  reason  for  carefully  distinguishing  between  questions 
of  fact  and  questions  of  desirsblllty  Is  that  often,  through  slopplness  or 
deviousneas,  one  type  of  question  la  substituted  for  the  other.  The  debate 
over  the  supersonic  transport,  for  example,  has  sometimes  focused  on  ths 
question  of  whether  It  would  generate  excessive  noise,  as  If  It  were  obvious 
that  If  It  did  not  generate  excessive  noise  it  would  be  a  good  thing. 
Similarly,  for  a  time  the  argument  over  the  shootings  at  Kent  State  Univer¬ 
sity  swirled  around  the  question  of  whether  the  National  Guardsmen  actually 
heard  cnlper  fire,  as  if  a  positive  answer  would  have  been  sufficient  to 
iastlfy  their  shooting  into  the  students  massed  in  front  of  them.  It  is 
in  this  way  that  questions  of  fact  are  sometimes  improperly  substituted  for, 
and  treated  as  surrogates  for,  questions  of  value. 


/ 
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3.3  Ben  Ring  the  Question,  Strategically 

Policy  makers  frequently  resist  policy  making.  Asked  "if  It  were 
true  that  it  was  his  ambition  to  go  down  in  history  as  the  Secretary  of 

State  who  solved  the  Berlin  crisis,"  Dean  Rusk  reportedly  replied  " 

,  I 

I’m  not  quite  that  vain.  But  I  do  want  to  go  down  in  history  as  one  of 

those  Secretaries  of  State  who  succeeded  in  pasting  the  Berlin  crisis  on 

to  his  successor!'  "  While  the  failure  to  face  up  to  hard  decisions 

often  does  reflect  an  abdication  of  responsibility,  it  is  sometimes 

justified.  Unnecessarily  impaling  decision-makers  on  the  horns  of 

dilemmas  wastes  good  talent.  Charles  Llndblom  has  been  one  of  the  few 

observers  to  plainly  acknowledge  that  "muddling  through"  is  sometimes 

positively  wise.  The  underlying  principle  is  simply  that  it  may  be  better 

to  make  no  decision  than  to  take  a  high  risk  of  making  a  bad  decision.  Of 

course,  'no  decision"  really  means  choosing  the  null  alternative.  If  there 

is  no  null  alternative  and  a  risky  decision  must  be  made,  the  choice  of  a 

less  consequential  alternative  may  help  to  assure  that  blunders  can  only 

be  small  ones. 

One  good  reason  for  refusing  to  choose  among  substantive  alternatives, 
and  instead  deferring  the  decision,  is  to  gain  time  to  obtain  new  information. 
Sometimes  delays  are  not  very  costly.  New  Information  may  be  expected  to 
present  Itself  to  the  decision-maker  with  the  passage  of  time,,  as  events  un¬ 
fold,  or  it  may  be  actively  pursued,  through  some  sort  of  research  process. 

It  may  be  anticipated  that  that  new  Information  will  clarify  the  nature  of 
the  alternatives  sufficiently  to  make  the  best  choice  obvious. 

Another  good  reason  for  deferring  decisions  is  simply  to  allow  more 


time  for  processing  the  information  the  analyst  already  has.  lore  thorough 


policy  analyses  take  more  time.  The  analyst  might  use  the  time  to  try  to 
develop  new  alternatives,  beyond  the  original  problematic  ones  he  had 
confronted.  A  skilled  decision-maker  usually  can  find  many  possible 
stopping  points  between  yes  and  no .  Instead  of  asking  whether  to  do  one 
/  thing  or  another,  he  may  find  It  possible  to  suggest  a  variety  of  Interest¬ 
ing  action  mixtures.  If  a  new  proposal  can  be  produced  which  Is  clearly 
superior  to  any  of  the  previous  considered  alternatives,  the  hard  question 
of  evaluation  will  have  been  elegantly  begged. 

Systematic  evaluation  procedures  can  be  avoided  by  avoiding  decisions, 
and  also  by  making  decisions  without  going  through  a  systematic  analysis.  In 
many  cases  the  differences  In  the  qualities  of  the  alternatives  will  be 
relatively  small,  so  that  the  risk  of  error  will  be  small.  It  Is  not  worth 
Investing  as  much  energy  Into  comparing  bicycles  being  considered  for  purchase 
as  It  Is  In  comparing  homes  being  considered  for  purchase.  Where  the  risk 
Is  small.  It  Is  simply  not  worthwhile  to  pursue  a  tedious  and  costly  analysis. 
In  foreign  policy  making,  however,  where  errors  can  be  extremely  costly, 
large  Investments  in  policy  analysis  often  are  warranted.  In  any  concrate 
decision  situation,  one  of  the  first  rlsk-balsnclng  questions  to  be  asked 
Is  whether  a  systemstlc  policy  analysis  should  be  conducted  at  all. 
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4.  SOME  SUGGESTED  EVALUATION  PROCEDURES 

4.1  Goals  and  Objectives 

It  Is  sometimes  suggested  that  one  of  th^  first  things  the  policy 
analyst  must  do  Is  to  define  his  objectives.  He  Is  asked  to  Identify  his 
target,  to  specify  where  he  wants  to  go.  Ilka  a  ganaral  mapping  out  the 
territory  ha  Intends  to  capture. 

Thoae  who  speculata  about  the  policy-making  process  have  always 

found  this  to  be  a  very  reasonabla-soundlng  suggestion,  bur.  practitioners 
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find  that  It  just  does  not  work  very  well.  A  decision-maker  facing  a 
crisis  situation,  for  example.  Is  not  likely  to  find  It  worthwhile  to  bagln 
his  policy  analysis  by  listing  a  series  of  objectlvas.  He  could  do  It, 
but  the  exercise  Is  not  likely  to  be  very  maanlnful  or  halpful.  The 
decision-maker  might  coma  to  gripe  with  hla  problem  more  effectively  If  he 
thinks  first  In  terms  of  the  different  kinds  of  responses  whlbh  are  available 
to  him,  examines  their  qualities,  and  then  makes  choices  where  choices  era 
required . 

One  reason  why  goals  and  objectives  are  not  very  useful  to  the  policy 
analyst  Is  that  they  are  usually  described  In  abstract  and  distant  terms,  so 
that  It  Is  difficult  to  decide  how  the  Immediately  available  concreta  action 
alternatives  differ  with  respect  to  thosa  goals.  Objectlvas  like  "the  defensa 
of  freedom"  or  "self-determination  for  all  peoplas"  hardly  provlda  adequate 
guidance  for  the  choices  of  action  encountered  on  a  day-to-day  basis. 

The  Imputation  of  goals  Is  useful  after  the  fact  to  help  under¬ 
stand  decisions  that  have  been  made,  but  thase  helpful  explanatory  constructs 
•nay  have  little  to  do  with  the  deliberative  processes  which  Impelled  the 
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act'ioas  in  tb«  first  pises.  As  Lassvell  observed,  "the  analysis  of  polities! 
rssolts  In  terse  of  certain  values  (like  4efereneet  safety,  in cose)  does  not 
inply  that  the  result  or  the  raises  are  consciously  sought."  Surely  few 
ststesssn  think  shout  the  notions!  interest  or  shout  the  pursuit  of  power  in 
the  oomIc  sense  that  Haas  Norfeotheu  snd  ethers  prepared  to  explain  past 
foreign  policies  do. 

I 

The  erslnstlon  of  altematlres  exelnslrely  in  terse  of  their  effec¬ 
tiveness  in  achieving  specified  object ires  is  very  narrow,  end  say  lead  to 
the  neglect  of  other  inpertsnt  considerations  like  costs  or  unintended 
cones fuences.  Of  course,  it  say  sonatinas  be  useful  to  state  objectives 
explicitly,  end  alternatives  should  be  cospared  in  terns  of  their  likelihood 
of  fulfilling  those  objectives.  The  point  is,  sisply,  that  thinking  in  these 
terns  does  not  bring  out  the  full  range  of  considerations  which  should  enter 
into  any  evaluation  of  alternatives. 

k,2  Criteria 

In  the  eyes  of  Xeheskel  Dror,  and  nsny  ethers  as  well,  ... 

In  principle  evaluation  involves  two  nain  steps: 

(l)  a  criterion  is  used  to  ascertain  the  actual  level 
or  quality  t including  both  quantitative  and  qualitative 
aspects  of  "quality")  of  a  process;  and  (2)  a  standards 
is  used  to  appraise  the  ascertained  quality.33 

Criteria  for  evaluation  nay  be  understood  as  dlnenslons  of  variation 
in  the  character  of  alternatives  which  have  clear  value  associations,  in 
the  senes  that  an  alternative  which  neasures  higher  on  one  dinenslon  is,  at 
least  in  that  respect,  preferred  to  ether  alternatives  which  neasure  lover 
on  that  dinenslon.  The  oost  of  an  ltan,  for  exanple,  night  be  regarded 
as  a  criterion  of  choice,  since  a  less  costly  iten  is  clearly  preferred 
to  a  wore  costly  version  of  that  sane  iten. 


These  diaenslons  can  also  be 
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■imply  nominal,  where  for  axample,  particular  colors  arc  praftrrod. 

For  any  suggested  criterion  to  serva  Its  function  in  evaluation.  It 
must  be  decidable  in  the  sense  that  it  can  be  clearly  determined  where  each 
alternative  stands  on  that  particular  dimension.  It  might  be  argued 
that,  say,  "likelihood  of  success"  would  be  a  good  baala  for  choosing  among 
alternatives,  but  the  criterion  la  not  useful  if  It  la  not  possible  to 
decide  which  of  the  elternatlves  la  higher  on  tliat  dimension.  Similarly, 
criteria  like  legality  or  morality  are  only  applicable  where  the  alternatives 
are  readily  differentiable  on  these  dimensions. 

Lovell,  like  many  other  observers,  suggests  that  any  valid  criterion 
for  choice  must  be  universally  acceptable,  universally  applicable,  and 
exclusive. 

The  requirement  that  criteria  must  be  universally  acceptable  suggests 
a  particular  concern  with  resolving  differences  among  decision-makers ,  that 
Is,  with  the  group  declelon-maklng  problem.  There  la  no  particular  reason 
why  the  criteria  used  by  the  individual  faced  with  a  problematic  choice 
should  necessarily  be  broadly  agreed  upon.  Consensus  Is  nice,  but  not 
necessary,  and  certainly  not  tha  best  Indicator  of  wisdom.  Inappropriate 
strictures  of  this  kind  can  seriously  confound  the  decision-maker's  talk. 

Insistence  on  universal,  or  at  least  general,  applicability  Implies 
a  search  for  criteria  which  usefully  distinguish  alternatives  in  a  very 
broad  variety  of  problem  situations.  In  this  conception,  criteria  would 
not  vary  from  one  decision  problsm  to  another ,  as  goals  or  objectives  would. 
It  seems  evident,  however,  that  since  the  signlflcent  dimensions  of  varia¬ 
tion  of  alternatives  are  very  different  In  different  situations.  It  will 
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not  be  possible  to  discover  sny  general  applicable  criteria  for  the  evalua¬ 
tion  of  alternatives. 

In  addition  to  suggesting  that  It  be  universally  applicable  and 
universally  agreed  upon,  Lovell  also  asked  that  any  criterion  for  choosing 
among  alternatives  should  be  exclusive,  in  the  sense  that  choices  should 
be  made  according  to  lagal  considerations  alona  or  according  to  moral 
considerations  alone .  Given  the  highly  multidimensional  nature  of  action 
alternatives  (apparent  from  any  thorough  description),  it  is  plain  that  no 
single  criterion  can  serve  as  an  adequate  guide  to  chloce.  The  only  measure 
which  might  conceivably  serve  in  this  way  would  be  a  construct  such  as 
utility  or  goodness  or  desirability  which,  by  its  definition,  comprehen¬ 
sively  subsumed  all  individual  dimensions  of  variation.  Herbert  Simon's 

generalized  criterion  of  efficiency  is  comprehensive  in  this  way,  but  ?nly 

36 

by  virtue  of  his  expanding  its  meaning  beyond  recognition.  Conceptual¬ 
izing  indicators  of  this  kind  is  not,  in  itself,  helpful,  since  it  is  not 
evident  how  the  utility  or  efficiency  of  alternatives  is  to  be  guaged. 

Indeed,  that  measurement  problem  is  simply  another  way  of  describing  the 
general  problem  of  evaluation  under  study  here. 

Criteria  like  legality  or  morality  may  nevertheless  be  useful, 
ilelevent  criteria  can  be  identified  through  an  exercise  In  which  hypothetical 
questions  based  on  the  ceteris  paribus,  or  "other  things  equal"  assumption 
are  posed.  If  two  alternatives  differed  only  in  the  degree  to  which  they 
were  legal,  and  the  decision-maker  would  prefer  the  more  legal  alternative, 
that  is  an  indication  that  legality  is,  for  him,  an  appropriate  criterion  to 
be  considered  in  evaluating  alternatives.  Any  such  distinction  which 
potentially  can,  under  certain  circumstances,  affect  the  choice  that  would 
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be  made,  can  be  regarded  aa  a  algnif leant  criterion  for  choice.  Legality 
may  not  be  the  criterion  by  which  cholcea  ahould  be  made,  but  It  la 
certainly  a  relevant  criterion.  The  quality  of  altcrnatlvea  is  partially 
determined  by  their  legal  character,  and  therefore  thia  variable  car  nake 
a  (inference  in  the  choice.  In  any  caae  in  which  the  alternatives 
differ  iu  their  legality,  that  difference  should  be  taken  Into  consider¬ 
ation. 

Used  with  care,  a  wide  variety  of  criteria  can  help  decision-makers 
to  form  their  cholcea  In  difficult  situations.  It  la  plain,  however, 
that  no  small  number  of  crltera  will  serve  aa  an  adequate  guide  to  choice 
for  moat  decision-makers  In  most  situations.  The  idea  of  criteria  may  be 
misleading  because  It  tends  to  narrow,  rather  than  expand,  the  range  of 
vision  of  the  analyst.  Potentially,  there  are  as  many  different  valid 
criteria  as  there  are  ways  of  distinguishing  among  alternatives.  An 
action  alternative  should  not  be  chosen  simply  because  it  has  some  good 
features,  and  should  not  be  rejected  simply  because  It  is  possible  to  point 
to  some  bad  features.  A  good  evaluative  analysis  Is  comprehensive,  and 
takes  Into  account  the  full  range  of  attributes  of  the  alternative  actions. 

4.3  Decomposition  Techniques 

John  Lovell  asks 

Are  there  any  objective,  or  universally  acceptable, 
criteria  by  which  one  may  measure  the  goodness  or 
badness,  the  desirability  or  undesirability  of 
foreign  policy  goals  and  means?  Or  are  the  norms 
of  evaluation  to  be  selected  or  rejected  merely  on 
the  basis  of  personal  preference?3' 

The  dichotomy  is  false  and  unhelpful.  It  is  not  true  that  if 
generally  acceptable  criteria  (or  goals)  cennot  be  found,  there  Is  nothing 
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left  to  guide  choice  but  taste  and  whim.  There  may  be  other  ways  to  help. 

A  number  of  considerations  always  enter  Into  the  determination  of 
preferences.  Usually  the  particular  elements  are  not  articulated,  but  are 

0 

viewed  as  a  unitary  gestalt,  without  differentiation.  This  la  typically 
the  case  for  non-problematic  decisions,  where  there  Is  no  felt  need  for 
probing  Investigation.  Where  decisions  are  difficult  to  make,  however, 
the  analyst  might  well  be  able  to  decompose  his  original  difficult  question 
Into  a  series  of  smaller  questions,  each  of  which  Is  easier  to  answer  than 
the  larger  decision  problem. 

Thus,  Instead  of  suggesting  specific  criteria,  one  can  suggest 
procedures  for  decomposing  the  larger  question.  For  example,  one  could 
suggest  doing  something  like  listing  the  alternatives  and  then  writing 
down  the  advantages  and  disadvantages  of  each  as  they  come  to  mind.  Cven 
very  simple  exercises  like  this  can  be  very  helpful.  They  can  clarify 
problems  sufficiently  so  that  appropriate  choices  become  apparent,  even 
without  further  sytematlc  and  explicit  analysis. 

Different  decision-makers  or  analysts  with  different  beliefs  will 
presumably  answer  the  smaller  questions  differently,  and  thus  may  produce 
different  responses  to  the  larger  question.  That  Is,  the  same  procedure 
may  lead  different  analysts  to  different  conclusions  because  they  may  use 
different  Inputs  Into  the  procedure.  The  procedure  may  nevertheless  be 
very  helpful  for  each  of  them.  Decomposition  procedures  for  evaluating 
action  alternatives  are  described  In  the  two  following  subsections. 

4.4  The  Expected  Value  Model 

In  discussing  goals,  objectives,  and  criteria,  it  was  assumed  that 
the  analyst  knew  everything  he  needed  to  know  about  the  alternatives  under 
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examination,  and  only  needed  a  way  to  clarify  his  thinking  about  them. 
Frequently,  however,  the  analyst  is  very  unsure  of  the  character  of  the 
alternatives  he  faces.  Two  very  different  kinds  of  uncertainty  can  be 
distinguished.  The  most  common  la  a  very  diffuse  kind  of  uncertainty,  where 
the  analyst  does  not  know  very  much  about  his  options,  lie  may  not  even  be 
able  to  form  clear  questions  about  them.  He  not  only  does  not  know  what 
consequences  they  would  yield,  he  hardly  knows  what  consequences  they  could 
yield.  His  understanding  of  the  situation  is  simply  very  fussy. 

Where  the  decision-theorist1 a  expected  value  model  is  supposed  to 
apply,  however,  another,  much  neater  kind  of  uncertainty  is  assumed  to  prevail. 
Here,  the  analyst  does  not  know  exactly  what  the  outcome  of  each  alternative 
action  would  be,  but  he  does  know  what  it  could  be.  The  possible  outcomes 
associated  with  each  alternative  are  small  In  number  and  clearly  distinguished 
from  one  another.  The  analyst  knows  the  quality  that  each  of  these  possible 
outcomes  would  have  if  It  were  obtained,  and  he  Is  also  able  to  judge  the 
likelihoods  with  which  each  action  would  lead  to  each  possible  outcome. 

The  objective  of  the  expected  value  model  is  to  show  how  this  "given" 
Information  can  be  combined  to  produce  a  reasonable  measure  of  the  quality,  or 
expected  value,  of  each  alternative  action  under  examination.  The  model 
asserts  that  the  expected  value  of  any  given  action  Is  equal  to  the  sum  of 
the  values  of  the  outcomes  to  which  It  might  possibly  lead,  each  discounted 
by  the  probability  of  Its  being  obtained.  In  other  words,  the  expected 
value  for  any  given  alternative  is  equal  to  the  product  of  the  probability 
measure  and  the  value  measure  for  each  possible  outcome,  summed  over  all 
the  possible  outcomes  for  that  alternative.  The  answer  is  strictly  true 
If  the  measure  of  value  and  of  the  probabilities  fulfill  the  strict  mathemat¬ 
ical  requirements  which  would  make  it  true.  If  rougher  measures  are  used. 
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the  answer  may  be  regarded  as  a  reasonable  estimate.  That  alternative  which 

has  the  highest  expected  value  is  the  one  that  is  best  and  should  be 

..  38 

chosen. 

The  model  is  very  logical,  but  it  is  not  very  useful  in  practice. 

It  applies  only  under  very  restricted  conditions,  and  it  demands  very 

special  kinds  of  information  as  input.  For  example,  the  formula  applies 

I 

only  if  the  possible  outcomes  are  clearly  enumerable,  and  only  if  they  are 
identified  in  a  manner  such  that  they  are  mutually  exclusive,  so  that  obtain¬ 
ing  one  strictly  precludes  the  possibility  of  obtaining  any  other.  This  la 
a  difficult  and  unnatural  way  of  describing  real  political  situations.  The 
method  is  Inconvenient  if  the  number  of  possible  outcomes  is  very  large,  or 
if  they  are  not  discretely  distinguishable.  Measures  of  the  estimated 
likelihoods  of  occurrence  and  of  the  values  of  the  possible  outcomes  which 
fulfill  the  mathematical  requirements  are  extremely  difficult  to  obtain. 

The  model  also  assumes  that  the  different  possible  outcomes  are 
already  evaluated  at  the  outset.  Although  it  does  not  say  how  that  is  to 
be  accomplished,  presumably  the  outcomes  are  to  be  evaluated  by  an  exten¬ 
sion  of  the  same  calculus,  whereby  the  value  of  a  given  possible  outcome  is 

determined  by  its  apparent  Instrumental  relation  to  still  more  remote 
possible  consequences. 

It  is  quite  reasonable  to  assert  that  the  quality  of  an  action  depends 
on  the  quality  of  the  outcomes  to  which  it  is  expected  to  lead,  especially 
if  outcome  is  construed  broadly  to  Include  all  kinds  of  "consequences” 
such  as  considerations  of  cost  and  otherwise  foregone  alternatives,  short- 
terms  responses  of  others,  long-term  reactions,  risks,  and  so  on.  Unfortun¬ 
ately,  however,  rather  than  bringing  out  the  rich  variety  of  possible  qualities 
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and  consequences  of  political  actions,  expected  value  analyses  tend  to 
lead  to  the  examination  of  only  a  few  narrow  catagorles  of  outcomes. 

The  most  important  failure  of  the  expected  value  approach  is  that 
It  does  not  address  itself  to  the  decision-maker's  problem.  Generally, 
if  a  political  analyst  knew  enough  about  the  problem  he  faced  to  formulate 

the  alternatives,  clearly  identify  the  possible  outcomes,  evaluate  them,  and 

I 

associate  unambiguous  likelihood  estimates  with  each  of  them,  he  would 
probably  know  enough  about  the  situation  to  know  which  alternative  would  be 
best,  without  bothering  to  carry  out  the  calculations  indicated  by  the 
formula.  Once  the  information  that  is  needed  to  begin  to  use  the  expected 
value  model  was  obtained,  the  problem  would  likely  be  solved.  The  problem 
lies  in  the  difficulty  of  obtaining  that  information.  The  analyst  typically 
begins  with  a  fuzzy,  diffuse  kind  of  uncertainty,  and  not  with  the  neatly 
ordered  kind  that  is  presumed  by  the  expected  value  model. 

A. 5  The  General  Ledger 

Evaluation  implies  comparative  analysis,  which  in  turn  Implies 
asking  the  same  questions  about  two  or  more  different  entities  and  then 
comparing  the  answers.  For  example,  If  the  analyst  Is  concerned  about 
meeting  certain  objectives,  he  must  ask  how  well  each  of  the  alternatives 
under  examination  would  meet  those  objectives,  and  then  compare  the  answers. 
If  the  answers  did  not  differ,  the  alternatives  could  not  be  distinguished 
on  those  grounds.  Thus,  evaluative  analysis  needs  to  focus  specifically  on 
the  differences  among  alternatives,  the  differentia. 

The  analysis  may  be  decomposed  into  a  primarily  empirical  or  factual 
part,  and  another,  more  clearly  evaluative  part.  Each  may  be  understood 
as  being  comprised  of  two  major  ateps.  In  the  first  step,  the  analyst 
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asks  how  the  alternatives  differ  In  fact,  past,  present,  or  anticipated; 
that  Is,  he  Identifies  the  apparently  significant  differences  among  the 
alternatives.  Second,  after  the  differentia  are  Identified  (named),  the 
analyst  elaborates  the  descriptive  characterization  of  those  differences. 

In  the  third  step,  the  analyst  examines  the  particular  differences  and 
evaluates  them,  deciding  which  tend  to  make  one  alternative  better,  and 
which  favor  the  choice  of  the  other  alternative.  Fourth,  and  finally, 
these  elementary  evaluative  judgments  are  then  combined  Into  an  overall 
evaluative  judgment  asserting  the  superiority  of  one  or  the  other  of  the 
alternatives. 

The  elementary  evaluative  judgments,  based  on  single  differentia, 
should  be  easier  to  form  than  the  composite  evaluation  because  each  of  them 
Is  based  on  a  single  dimension  of  variation. 

The  analysis  of  a  decision  problem  can  be  laid  out  In  the  form  of  a 
general  ledger,  as  shown  In  Table  1.  It  is  described  as  a  ledger 
simply  because  It  provides  a  way  to  account  for  credits  and  debits, 
assets  and  liabilities,  advantages  and  disadvantages,  costs  and  benefits. 
F.ach  row  corresponds  to  one  feature,  that  Is,  to  one  question,  one  basis 
for  comparison,  one  variable  or  dimension  which  characterizes,  and 
possibly  distinguishes  the  alternatives.  A  pair  of  columns  is  associated 
with  each  alternative.  The  first  column  of  the  pair  provides  a  place  for 
the  analyst's  best  understanding  of  the  factually  correct  answer  to  the 
question  posed  In  that  particular  row.  In  practice,  of  course,  the  full 
answer  will  have  to  be  provided  elsewhere,  in  generous  prose,  and  no  more 
than  a  few  summarizing  words,  serving  as  reminders,  can  be  Included  In 
the  table  Itself.  The  second  column  of  the  pair  summarily  records  the 
analyst's  evaluative  response  to  the,  facts  as  he  understands  them. 
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To  Illustrate,  suppose  a  policy  analyst  concerned  with  United 
States  action  with  respect  to  the  Palestinians  has  already  decided  to 
recommend  that  the  United  States  should  help  those  Palestinians  who  wish 
to  do  so  to  immigrate  to  the  United  States.  The  question  night  then  arise 
as  to  whether  the  United  States  government  should  go  so  far  as  to  provide 

i 

the  transportation.  The  action  alternatives,  then,  are  for  the  United 
States  government  to  provide  the  transportation,  or  for  it  not  to  provide 
the  transportation.  For  the  sake  of  the  illustration,  It  is  assumed  that  a 
choice  must  be  made  between  these  two  options. 

After  describing  these  action  possibilities  In  some  detail,  the 
analyst's  next  step  would  be  to  try  to  think  of  and  name  the  kinds  of 
features  which  determine  their  qualities.  He  does  not  have  to  worry  in 
advance  about  whether  they  will,  In  fact,  distinguish  between  the  alterna¬ 
tives.  The  features  might  be  like  those  listed  In  Table  1.  The  analyst  may 
not  know  the  answers  to  these  questions  at  the  outset,  but  formulating  them 

will  help  him  to  decide  what  Information  matters,  and  will  help  him  to  guide 
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his  research.  The  answers  then  constitute  parts  of  his  empirical  charac¬ 
terization  of  the  alternatives.  Descriptive  terms  serving  as  reminders  of 
those  detailed  characterizations  may  be  Included  in  the  ledger  table. 

The  analyst  Is  free  to  include  In  the  list  whatever  questions  seem 
to  him  to  be  significant.  If  he  thinks  his  choice  might  be  affected  by, 
say,  Chinese  reactions,  then  that  item  should  be  included  In  the  list.  If 
the  questions  that  are  first  posed  are  found  to  be  difficult  to  answer,  the 
analyst  may  sometimes  find  It  useful  to  refine  or  decompose  them  further. 
Some  of  the  questions,  like  the  one  about  moral  quality,  may  not  be  of  the 
sort  that  are  normally  regarded  as  purely  empirical  or  factual.  That  does 
not  matter.  Any  variable  on  which  the  alternatives  can  be  characterized 
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and  which  should  be  considered  In  the  overall  analysis  may  be  Incorporated 
within  this  scheme,  by  these  procedures,  whether  or  not  the  question  is  a 
purely  empirical  one. 

Once  the  information  base  is  established,  the  analyst  can  then 
proceed  to  form  evaluative  judgments  on  each  of  the  elements.  Of  course, 

J 

in  some  Instances  a  knowledgable  analyst  may  be  able  to  form  clear 

I 

evaluative  judgments  without  first  developing  detailed  and  explicit 
empirical  characterizations  of  the  proposed  actions.  In  any  case,  the 
evaluations  will  reflect  the  analyst's  response  to  what  he  understands 
to  be  the  facts.  Ills  evaluations  of  each  of  the  features  may  be  recorded 
simply  as  judgments  of  ‘'good"  and  "bad,"  or  he  might  use  a  more  elaborate 
judgmental  scale,  possibly  with  higher  positive  numbers  indicating  greater 
goodness'  and  higher  negative  numbers  indicating  greater  "badness." 

A  similar  "balance  sheet"  method  for  the  comparative  analysis  of 

policy  alternatives  has  been  developed  and  illustrated  hy  Halph  White 
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in  a  study  of  five  different  Vietnam  policies.  The  policies  were 
measured  against  ten  different  "values"  like  "avoiding  World  War  III  in 
the  long  run,"  "avoiding  the  domino  tendency,"  and  "ending  the  war 
quickly."  While  these  values  were  not  chosen  blindly,  as  a  priori 
criteria,  they  did  not  go  to  that  other  extreme,  advocated  here,  of 
attempting  to  take  into  account  all  significant  features  by  which  the 
particular  alternatives  can  be  distinguished. 

The  ledger  idea  has  deep  historical  roots,  tracing  back  at  least 
to  Jeremy  Bentham's  'fcllcific  calculus"  calling  for  the  balancing  of 
pleasures  and  pains.  It  can  also  he  viewed  as  a  generalization  of  the 
expected  value  model.  In  the  expected  value  analysis,  the  distinctive 
features  are  the  actions'  possible  outcomes  ,  each  having  specifies  values. 
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and  che  only  empirical  characterizations  of  concern  are  the  likelihoods 
with  which  chese  possible  outcomes  will  be  obtained  as  a  result  of 
selection  of  a  particular  action  alternative. 

After  evaluations  on  each  of  the  relevant  features  are  determined* 

.  I 

the  analyst  must  somehow  put  all  that  information  together  to  determine 
which  of  the  alternatives,  taken  as  a  whole,  would  bo  best  to  choose.  The 
natural  inclination  is  to  calculate  some  sort  of  sum  for  each  evaluation 
column,  with  the  more  important  features  given  extra  weight.  The  more 
desirable  alternative  would  then  be  that  one  which  had  the  higher  overall 
measure  of  quality.  This  seems  to  capture  the  way  in  which  people 
ordinarily  think  of  analysing  alternatives. 

Sometimes,  however,  thinking  in  these  terms  can  lead  to  careless 
propositions.  For  example,  it  is  sometimes  suggested  that  one  should  take 
an  action  if  its  advantages  outweigh  its  disadvantages.  It  should  be  clear 
that  this  would  be  true  only  if  the  only  alternative  against  which  it  was 
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to  be  compared  was  on  whose  net  value  was  counted  as  sero.  There  may 
actually  be  a  number  of  mutually  exclusive  alternatives,  all  of  which 
have  a  net  positive  quality,  in  which  case  the  rule  would  provide  no 
guidance  at  all.  .Sometimes  one  may  have  to  choose  the  best  out  of  s  had 
lot  of  alternatives,  where  positive  features  arc  outweighed  by  negative 
features  in  every  case.  It  may  be  wiser  to  take  an  action  than  not  to 
take  it,  but  it  does  not  always  follow  that  it  would  therefore  be  wise 
to  take  that  action.  This  is  so  simply  because  there  may  be  other  alterna¬ 
tive  actions  which  would  be  even  better.  The  good  can  be  the  enemy  of  the 
best. 

The  suggestion  that  an  alternative  should  be  chosen  if  its  assets 
outweigh  its  liabilities  is  misleading  because  It  falls  to  take  into  account 
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the  essentially  comparative  nature  of  the  evaluation  process.  The  analyst 
should  not  simply  compute  the  (possibly  weighted)  sum  for  a  single  column 
and  examine  it  to  determine  whether  it  is  positive  or  negative.  Me  should 
compare  that  measure  of  the  quality  of  one  alternative  with  the  corresponding 
measure  of  quality  for  that  alternative's  competitors. 

Each  of  the  individual,  elementary  evaluative  assessments  should  be 
made  in  relation  to  some  clear  reference.  If  several  substantive  alternatives 
are  under  examination,  it  is  natural  to  choose  the  null  alternative  as  the 
common  reference.  The  assessments  would  then  show  the  decree  to  which  each 
of  the  substantive  alternatives  were  seen  as  better  or  worse  than  the  null 
alternative  on  each  of  the  enumerated  dimensions.  Any  other  alternative 
could  just  as  well  be  used  as  the  reference  standard. 

The  essentially  comparative  nature  of  the  evaluation  process  is 
fully  acknowledged  if.  Instead  of  using  an  external  reference  standard 
of  measurement,  the  alternatives  under  examination  are  compared  directly 
with  one  another.  Contrary  to  the  suggestion  of  *)ror,  lechan  and  others, 
there  Is  really  no  need  for  a  standard  independent  of  the  alternatives 
themselves.  Table  2  shows  a  revised  general  ledger  format  which  brings 
out  the  differences  directly.  Here,  only  one  column  Is  provided  for 
empirical  information,  and  it  asks  how  the  alternatives  differ  with  respect 
to  each  of  the  particular  features.  The  analyst,  sensitized  to  this  concern 
for  differences,  no  longe**  needs  to  expend  his  research  effort  on  total  or 
absolute  characterizations  of  the  alternatives.  It  is  on  the  basis  of 
these  c’lfferenccs  in  the  character  of  the  alternatives  that  comparative 
elementary  evaluations  are  made.  At  Its  simplest,  the  evaluation  question 
on  each  feature  asks:  If  the  alternatives  differed  only  in  this  respect, 
which  of  them  would  be  preferred?  Instead  of  forming  some  absolute,  overall 
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Table  2.  The  Revised  General  Ledger 
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measure  of  Che  qualicy  of  each  of  Che  alternatives  ami  then  relating  those 
measures,  now  the  individual,  elemental  measures  are  compared  across  the 
alternatives. 

Whether  the  basic  or  the  revised  general  ledger  is  used,  it  is  still 
necessary  to  find  appropriate  measures  for  the  elementary  evaluations. 

\ppropriate''  should  be  taken  to  mean  adequate  to  the  task,  and  not  neces¬ 
sarily  as  a  requirement  for  mathematical  elegance.  It  may  sometimes  be 
possible  to  get  by  with  very  rough  elemental  evaluations,  distinguished 
into  categories  no  more  refined  than,  say,  "good"  and  "bad,"  or  "better" 
and  "worse.  '  These  will  be  adequate  where,  for  example,  it  is  found  that 
"goods"  consistently  fall  on  one  side  of  the  ledger,  while  the  'bads' 
fall  on  the  other  side,  itore  sophisticated  measures,  generally  more 
difficult  to  obtain,  can  be  Introduced  if  and  when  the  ’cheaper"  ones 
prove  to  be  inadequate. 

Together  with  the  measurement  problem,  there  also  still  remains 
the  analytic  problem  of  determining  appropriate  rules  by  which  the 
elementary  evaluative  judgments  should  be  combined  to  form  the  larger, 
composite  evaluation.  One  might  try  to  associate  weights  with  each  of 
the  particular  features  to  reflect  their  relative  importance,  and  then 
compute  column  sums.  This  may  be  impractical,  however,  because  the 
effort  it  would  entail  would  be  too  great,  and  would  draw  energies  away 
from  the  particular  decision  problem  at  hand.  Another,  interesting, 
approach  might  be  to  simply  let  the  magnitude  of  each  elemental  measure 
reflect  an  implicit  weighting,  so  that  the  more  important  dimensions 
hud  higher  positive  or  negative  measures  associated  with  them.  The 
values  of  these  measures  might  be  determined  by  successive  approximation. 
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as  a  result  of  posing  a  series  of  questions  as  to  which  choice  would  he 
made  with  given  hypothetical  changes  In  the  nature  of  the  alternatives. 

For  example,  the  analyst  might  ask  what  choice  he  would  make  if  the 
alternatives  differed  only  with  respect  to  feature  I,  only  with  respect 
to  II,  only  with  respect  to  II  and  III,  only  with  respect  to  I  and  VI, 
and  so  on,  and  continuously  adjust  the  numbers  in  the  cells  of  the 
ledger  until  they  reflected  his  answers  to  all  these  hypothetical 
questions.^ 

In  practice,  however,  it  may  not  be  important  to  have  exact  measures 
and  perfectly  clear,  logical  rules  of  combination.  Reasonable-sounding 
answers  to  these  questions  can  be  given,  but  they  all  have  serious 
deficiencies  in  practice.  This  essay  intentionally  stops  short  of  trying 
to  provide  generally  satisfactory  answers.  The  argument  here  is  simply 
that  analyzing  decision  problems  in  the  revised  general  ledger  format 
that  has  been  described  will  itself  often  clarify  problematic  decision 
situations  sufficiently  to  make  wise  choices  obvious.  The  policy  analyst 
may  use  the  simplest  scheme  sufficient  to  his  task,  and  that  will  often 
mean  that  he  can  stop  before  encountering  rigorous  measurement  problems 
or  the  formal  analytic  problem  of  composition. 

4.6  The  Pair-Wise  Evaluation  Strategy 

It  is  not  the  purpose  of  this  essay  to  study  the  policy  analysis 
process  as  a  whole,  but  the  revised  general  ledger  technique  has  certain 
implications  for  that  process  which  should  be  drawn  out.  It  may  be 
objected  that  that  evaluation  procedure  is  well  suited  to  the  comparison 
of  pairs  of  alternatives,  but  in  realistic  situations  it  is  often  necessary 
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to  examine  much  larger  numbers  of  alternatives.  Kxccpt  for  certain  mathe¬ 
matically  anomolous  situations,  however,  (those  In  which  preference 
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orderings  arc  intransitive),  pair-wise  analysis  is  entirely  appropriate. 

I'he  analyst  can  zero-in  on  the  one  best  action  alternative  by  comparing 

two  at  a  time,  lie  can  begin  by  comparing  any  pair  of  the  alternatives. 

! 

The  inferior  one,  the  loser"  in  th.it  comparison,  can  he  rejected  alto¬ 
gether  it  does  not  have  to  be  compared  with  any  of  the  other  alternatives. 

The  winner  '  is  then  compared  with  another  alternative,  and  the  'loser  ' 
of  that  second  round  of  evaluation  can  then  be  rejected.  This  step-wise 
comparison  of  pairs  is  continued  until  the  one  best  alternative  is 
ldentlf  ied.^ 

This  procedure  makes  use  of  the  fact  that  there  is  no  need  to  locate 
all  the  steps  in  the  rank-ordered  preference  ladder  if  the  only  real  require¬ 
ment  is  to  locate  the  top  rung.  The  full  preference  ordering  does  not  have 
to  be  determined,  so  the  total  number  of  comparisons  that  have  to  be  made 
is  sharply  reduced.  With  the  pair-wise  evaluation  strategy,  the  best 
alternative  is  located  rapidly  and  efficiently. 

The  strategy  also  suggest  a  guideline  for  the  investment  of  effort 
into  the  articulation  of  alternatives.  The  rule  is  simply  that  an 
alternative  does  not  have  to  be  developed  in  detail  unless  it  has  some 
reasonable  chance  of  showing  Itself  to  be  superior  to  the  current,  tenta¬ 
tively  chosen,  best  alternative.  According  to  the  procedure  described 
earlier,  the  policy  analyst  drew  up  a  list  of  alternatives,  then  described 
them  in  detail,  and  then  evaluated  them.  The  policy  analyst  could  Instead 
proceed  by  first  fully  articulating  one  promising  course  of  action,  and 
making  a  tentative  commitment  to  choosing  it.  lie  then  tries  to  think  up 


4  9. 


or  find  some  other  course  of  action  (possibly  a  minor  variation  on  the 
one  in  hand)  which  would  he  better  than  the  tentatively  chosen  one.  If 
lie  finds  any  candidate  which  apparently  would  have  a  hope  of  shovin'’  itself 
to  be  better  than  the  current  tentative  best  one,  he  develops  it  in  more 
detail,  and  then  carries  out  a  systematic  evaiuat ive  analysis,  comparing 
the  new  candidate  with  the  previous,  tentative  winner.  If  that  newly 
developed  candidate  is  found  to  be  superior,  he  then  makes  a  tentative 
commitment  to  choosing  it.  This  process  of  developing  and  systematically 
evaluating  new,  promising  candidates  is  continued  until  no  new  alternative 
can  be  found  to  challenge  the  current  tentative  winner,  at  which  tine  the 
evaluation  procedure  is  concluded.  With  this  evaluation  strategy,  rather 
than  being  spread  all  over  the  field,  the  energy  available  for  articulating 
alternatives  is  invested  only  where  it  promises  to  provide  the  highest 
possible  benefit .  Furthermore ,  this  openness,  this  constant  probing,  helps 
to  protect  the  analyst  against  premature  closure.  With  this  scheme,  he 
will  be  less  likely  to  deceive  himself  into  believing  that  a  particular 
alternative  is  the  best  possible  when  he  has  only  shown  that  it  happens 
to  be  better  than  some  "straw-man"  alternative. 

4.7  Conclusion 

The  literature  on  prescriptive  foreign  policy  analysis  is  extra¬ 
ordinarily  meager.  The  Insights  there  are  camouflaged  by  a  great  variety 
of  misconceptions  and  inadeaute  conceptions  of  the  way  in  which  wise 
choices  of  action  should  be  distinguished  from  unwise  ones.  The  purpose 
of  this  essay  has  been  to  help  generate  a  sensitivity  to  t!.e  problems 
encountered  in  the  evaluation  of  policy  alternatives,  and  to  suggest 
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some  specif  if  guldel  i  nos  for  forming  evaluations. 

Schemes  which  suggest  analysis  in  terms  of  particular  goals  or 
criteria  or  outcomes  tend  to  narrow  rather  than  broaden  the  policy 
analyst's  range  of  vision.  In  contrast,  the  use  of  an  account-book 
procedure  permits  the  full  range  of  positive  and  negative  qualities 
of  the  options  to  be  systematically  examined.  A  suggested  revised  general 
ledger  format  focuses  directly  on  the  comparative  analyses  that  are  required 
and  this,  in  turn,  forms  the  basis  of  a  highly  efficient  evaluation  strategy 

The  analysis  process  was  described  in  terns  of  operations  undertaken 
by  a  single  individual  in  response  to  a  problem  felt  by  a  single  Individual. 
The  assertion  of  the  natural  priority  of  individual  over  group  decision¬ 
making  should  not  be  taken  to  mean  that  individual  preferences  should 
always  be  firmly  established  before  entering  into  any  social  encounter. 
f»verly  rigid  determination  of  Individual  preferences  may  seriously  Impede 
group  decision-making ,  to  the  disadvantage  of  all  concerned.  On  the  other 
hand,  group  interaction  may  frequently  be  of  great  help  to  the  individual 
in  analyzing  his  decision  problem.  Rather  than  simply  deferring  to  the 
group,  the  individual  may  use  the  group  to  help  him  to  understand  the 
problem  and  to  come  to  his  own  decision. 

decomposition  schemes  like  those  described  here  can  be  especially 
useful  for  group  decision-making.  If  different  decision-makers  disagree 
on  what  should  be  done  in  a  particular  case,  a  systematic  analytic 
procedure  can  help  to  make  the  bases  for  the  decisions  explicit.  It  can 
help  the  Individuals  to  identify  the  sources  of  their  differences,  and 
this  in  turn  can  help  them  to  find  ways  to  reconcile  those  differences. 

.tuch  of  the  evaluation  work  required  in  the  larger  policy  analysis 
process  can  be  accomplished  perfectly  well  without  systematic,  explicit 


procedures.  Hard  questions  of  evaluation,  as  distinguished  from  questions 
of  fact,  for  example,  probably  arise  only  rarely,  and  when  they  do  arise. 
It  may  be  possible,  and  wise,  to  avoid  them  in  one  way  or  another.  .  ever- 
theless,  it  is  Important  that  the  nature  of  the  evaluation  process,  the 
ways  In  which  it  should  be  conducted,  and  the  ways  In  which  It  should  not 
be  conducted,  are  well  understood  if  the  selection  of  policy  alternatives 
is  to  be  soundly  based.  In  the  realm  of  foreign  policy,  In  particular, 
the  Investment  of  effort  would  surely  be  worthwhile. 
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when  (on  p.  30)  he  skips  from  the  perspective  of  a  decision-maker  evaluating 
action  alternatives  to  that  of  a  detached  critic  observing  a  decision-maker. 
At  the  same  time,  he  narrows  the  variety  of  qualities  to  be  considered 
down  to  those  of  consequence  for  human  welfare,  presumably  with  the 
intention  of  examining  more  purely  ethical  questions.  In  my  understanding, 
ethical  questions  are  particularly  concerned  with  the  effects  a  decision 
may  have  on  the  welfare  of  others.  No  ethical  questions  arise  for  a 
decision-maker  contemplating  action  alternatives  which  would  affect 
no  person  other  than  himself,  but  value  questions  still  arise  for  him 
in  choosing  among  the  alternatives.  Meehan  seems  to  mix  the  problem 
of  Judging  the  moral  character  of  an  actor  vith  the  decision-maker's 
task  of  Judging  the  overall  qualities  of  alternative  courses  of  action. 

?6 

Davis  Bobrov  (ed. ) ,  Weapons  Systems  Decisions:  Pollt*~al  and 
Psychological  Perspectives  on  Continental  Defense .  New  York:  Praeger,  1969. 


27 

A  systematic  procedure  for  moving  back  and  forth  between  higher 
and  lover  levels  of  abstraction,  described  as  "mixed-scanning,"  has  been 
developed  In  Amitai  Etzioni,  The  Active  Society:  A  Theory  of  Societal  and 
Political  Processes,  New  York:  Free  Press,  i960,  pp.  2&2-309. 


2®Meehan,  Value  Judgment  and  Social  Science,  p.  3;  also  p.  U8. 


29 

David  Braybrooke  and  Charles  E.  Lindblom,  A  Strategy  of  Decision: 
Policy  Evaluation  as  a  Social  Process,  New  York :  Free  Press ,  1953,  p.  86. 


^°Thi s  distinction  between  the  fact  and  value  aspects  of  evaluation 
corresponds  to  the  distinction  between  affective  and  belief  components  of 
attitudes  in  social  psychology.  Cf.  Martin  Fishbein  (ed.).  Raidings  in 
Attitude  Theory  and  Measurement ,  Nev  York:  Wiley,  1967.  Fishbein ' s  suggested 
rule  of  composition  by  siasaation  of  products  of  affect  and  belief  components 
(pp.  39*»,  *»37 )  may  be  viewed  as  a  generalisation  of  the  expected  value  model 
to  be  described  later.  The  formulation  is  offered  as  a  basis  for  explanation, 
but  it  could  reasonably  be  suggested  as  a  basis  for  prescription. 


^Harold  D.  Las swell,  Politics:  Who  Gets  What.  When.  How.  Nev  York: 
Meridian,  1958,  p.  189  (postscript). 
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32 

Hllsman,  To  love  \  .\ation,  p.  41. 

33 

Ct.  Charles  .1.  Hitch,  'On  the  Choice  of  Objectives  in  Systems 
Studies,''  in  Donald  l‘.  Kckman,  (ed.),  System r  :  'e search  and  ’’eslgn, 

Tew  York:  Wiley,  pp.  43-51;  Aaron  Wildavaky,  'The  Political  Economy  of 
efficiency:  Cost  Benefit  Analysis,  Systems  Analysis,  and  Program 
Budgeting,  '  in  Austin  tanney  (ed.),  Political  Science  and  Public  Policy, 
Chicago:  'tarkham,  196H,  especially  pp.  64-67;  l.indhlom,  "The  Science  of 
iuddling  Through. 

34 

Lasswell,  Politics,  p.  26. 

35 

Yehezkel  oror ,  Public  Policymaking  Reexamined ,  San  Francisco: 
Chandler,  1968,  p.  25.  A  good  discussion  of  the  selection  and  use  of 
criteria  may  be  found  in  Charles  J.  Hitch  and  Roland  N.  McKean,  The 
Economics  of  Defense  in  the  Nuclear  Age,  Hew  York:  Atheneun,  1967, 
pp.  158-205. 

36 

Herbert  A.  Simon,  Administrative  Behavior ,  Second  edition, 

Sew  York:  Free  Press,  1957,  pp.  178-186. 

37 

Lovell,  Foreign  Policy  in  Perspective,  pp.  2B9-290. 

38 

For  thorough  expositions  of  the  expected  value  model,  cf.  Robert 
Schlalfer,  Introduction  to  Statistics  for  Business  Decisions,  New  York: 
McGraw-Hill,  1961;  Howard  Ralffa,  Decision  Analysis:  Introductory  Lectures 
on  Choices  Under  Uncertainty.  Reading,  Mass.:  Addison-Wesley,  1968. 
Limitations  on  the  applicability  of  the  expected  value  model  and  others 
li-c  it  are  discussed  in  Braybrooke  and  Lindblom,  A  Strategy  of  Decision, 
pp.  21-33;  Giinter  Manges,  "The  Suitability  of  the  General  Decision  Model 
for  Operational  Applications  in  the  Social  Sciences,"  in  J.  R.  Lawrence, 
(ed.),  Operational  Research  and  the  Social  Sciences,  London:  Tavistock, 
1966,  pp.  565-577;  Phllburn  Ratoosh,  ''Defense  Decision-Making:  Cost 
Effectiveness  Models  and  Rationality,'1  in  Bobrow,  Weapons  System  Decisions. 

39 

A  method  for  assessing  likely  reactions  to  particular  proposals  is 
suggested  in  my  paper,  "Foreign  Policy  Analysis:  Middle  liast,"  Peace 
Research  Society:  Panera .  Vol.  XIV  (1970),  pp.  95-112. 

40 

Ralph  »(.  White,  Nobody  Wanted  War:  Misperception  in  Vietnam  and 
Jther  Wars,  -lew  York:  Doubleday,  1970,  Appendix,  pp.  339-358. 

41 

A  similar  process  is  Illustrated  in  Blechman,  "The  Quantitative 
Evaluation  . . . . ‘ 

42 

On  the  problem  of  intransitive  preference,  cf.  Luce  and  Ralffa, 
Games  and  Decisions .  p.  25. 
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43 

This  strategy  should  not  be  confused  with  the  search-by-dichotomy 
procedure  suggested  by  Information  theory,  a  procedure  designed  for  moving 
from  abstract,  inclusive  categories  down  to  specifics  in  a  successive 
branching  process.  The  pair-wise  strategy  is  intended  for  operations  at 
a  single  level  of  abstraction.  Applied  at  a  single  level,  search-by- 
dlchotomy  calls  for  dividing  the  alternatives  Into  two  subgroups,  and 
then  deciding  which  of  them  would  be  likely  to  contain  the  best  alterna¬ 
tive.  Then  that  selected  subgroup  would  be  divided  again  Jntc  two  parts, 
and  then  one  of  those  would  be  selected.  This  process  of  division  and 
selection  would  continue  until  one  alternative  was  chosen.  The  procedure 
is  useful  when  there  Is  some  non-arbltrary  basis,  such  as  strong  similarity, 
for  clustering  alternatives  within  given  subgroups.  A  procedure  like  this 
is  used  intuitively  when  moving  from  abstract,  general  description*  of 
proposed  courses  of  action  down  to  specifics.  On  search-by-dlchotomy, 
cf.  Stafford  Beer,  Cybernetics  and  Management,  i low  York:  Wiley,  1959,  pp.  55- 
56,  02-87. 
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